KR101988157B1 - Timeline synchronization method and apparatus of video clips using sound waveform - Google Patents

Abstract

A method and an apparatus for synchronizing a time axis among multiple videos by using a sound waveform are disclosed. According to an embodiment of the present invention, the method for synchronizing a time axis comprises the steps of: receiving multiple videos; extracting multiple sound waveforms respectively corresponding to the multiple videos from the multiple videos to generate multiple waveform images; and synchronizing the multiple videos based on similarity among the multiple waveform images.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a time synchronization method and apparatus for a plurality of moving pictures using a sound waveform,

The following embodiments relate to a method and apparatus for time axis synchronization between multiple moving images using a sound waveform.

Recently, ordinary users can easily create panoramic images using smart phones. However, there are still difficulties in making panorama videos using smart phones.

One of these difficulties is that when shooting images at the same time, it is necessary to make panorama videos by shooting in various angles with various smart phones, because it is difficult to synchronize the time of each video due to hardware differences in smartphone.

In order to overcome the problem that it is difficult to obtain the same time between multiple videos due to differences between hardware, methods of synchronizing time axes among various videos using image information have been studied.

In the case of an image information based time axis synchronization method using color information or image features, a large amount of data is required because of a large data size, and it takes a long time or a time synchronization accuracy is poor.

Video consists of video and audio. In other words, audio also means that it can be used for time synchronization between moving images. Common audio features include audio fingerprinting, spectrograms, and sound waveforms.

Among them, the sound waveform has an advantage that it is easy to understand and easy to handle.

Embodiments can provide a technique of synchronizing the time axis between a plurality of moving images using a sound waveform.

According to an embodiment, there is provided a time synchronization method including: receiving a plurality of moving images; extracting a plurality of sound waveforms corresponding to each of the plurality of moving images from the plurality of moving images to generate a plurality of waveform images; And synchronizing the plurality of moving images based on the degree of similarity between the plurality of waveform images.

The plurality of waveform images may be a binary image having a length corresponding to twice the frame number of each of the plurality of moving images.

The generating may include generating the plurality of waveform images from the plurality of sound waveforms using Fast Forward Moving Pictures Expert Group (FFmpeg).

The synchronizing step may include performing a comparison operation between the plurality of waveform images to calculate the similarity, and synchronizing the time axes of the plurality of moving images based on the similarity.

The step of synchronizing the time axis may include the step of matching the time axis to a time point at which the degree of similarity is the highest.

Wherein the step of calculating comprises: classifying the plurality of waveform images into a reference image and a comparison object based on a length of the plurality of waveform images; selecting a portion of the reference image; And calculating the similarity by performing a comparison operation on the comparison object image.

The classifying step may include setting a waveform image having the shortest length among the plurality of waveform images as the reference image.

The selecting may comprise selecting half of the reference image as a portion of the reference image.

The step of calculating the degree of similarity may include a step of selecting a part of the comparison object image and a step of calculating an XOR (Exclusive-OR) operation between a part of the reference image and a part of the comparison image to calculate the degree of similarity .

A time synchronization apparatus according to an embodiment includes a receiver for receiving a plurality of moving pictures and a controller for synchronizing the plurality of moving pictures based on a similarity degree of a plurality of sound waveforms extracted from the plurality of moving pictures.

The controller includes a waveform image generator for extracting a plurality of sound waveforms corresponding to each of the plurality of moving images from the plurality of moving images to generate a plurality of waveform images; And a synchronizer for synchronizing the plurality of moving images.

The plurality of waveform images may be a binary image having a length corresponding to twice the frame number of each of the plurality of moving images.

The waveform image generator may generate the plurality of waveform images from the plurality of sound waveforms using Fast Forward Moving Pictures Expert Group (FFmpeg).

The synchronizer may include a similarity calculator for performing a comparison operation between the plurality of waveform images to calculate the similarity, and a time axis adjuster for synchronizing the time axis of the plurality of moving images based on the similarity.

The time axis adjuster may match the time axis with a time point at which the degree of similarity is the highest.

Wherein the similarity calculator classifies the plurality of waveform images into a reference image and a comparison object based on a length of the plurality of waveform images, selects a part of the reference image, The degree of similarity can be calculated.

The similarity calculator may set a waveform image having the shortest length among the plurality of waveform images as the reference image.

The similarity calculator may select half of the reference image as a part of the reference image.

The similarity calculator may calculate the degree of similarity by performing a XOR (Exclusive-OR) operation between a part of the reference image and a part of the comparison image by selecting a part of the comparison image.

Figure 1 shows a schematic block diagram of a time synchronization device according to one embodiment.
Fig. 2 shows a schematic block diagram of the controller shown in Fig.
Figure 3 shows a schematic block diagram of the synchronizer shown in Figure 2;
4 shows an example of the operation of the time synchronization apparatus shown in Figs.
5 shows an example of an operation in which a synchronizer performs synchronization.
6A shows an example of a time-base synchronization result based on a sound waveform.
6B shows an example of the time-base synchronization result based on the image information.
7 shows a flowchart of a time synchronization method according to one embodiment.

In the following, embodiments will be described in detail with reference to the accompanying drawings. However, various modifications may be made in the embodiments, and the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and alternatives to the embodiments are included in the scope of the right.

The terms used in the examples are used for descriptive purposes only and are not to be construed as limiting. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The terms first, second, or the like may be used to describe various elements, but the elements should not be limited by terms. The terms may be named for the purpose of distinguishing one element from another, for example, without departing from the scope of the right according to the concept of the embodiment, the first element being referred to as the second element, The second component may also be referred to as a first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this embodiment belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. In the following description of the embodiments, a detailed description of related arts will be omitted if it is determined that the gist of the embodiments may be unnecessarily blurred.

Figure 1 shows a schematic block diagram of a time synchronization device according to one embodiment.

Referring to FIG. 1, the time synchronization apparatus 10 may synchronize a plurality of moving pictures using a sound waveform of moving pictures. The time synchronization device 10 can synchronize the time axis of a plurality of moving pictures using a sound waveform, so that the time synchronization device 10 can be faster and more accurate than the conventional image-based synchronization method.

Conventional image-based synchronization method can not be used when there is no movement in the moving image, and synchronization accuracy may be lowered when the movement is too fast. In addition, since the conventional image-based synchronization method extracts features from image information, it may take a long time without using a GPU (Graphics Processing Unit).

The time axis synchronization apparatus 10 can generate a panorama moving image by synchronizing moving images captured by a plurality of cameras at the same time. The time base synchronization apparatus 10 can synchronize a plurality of videos by searching for the same time zone from a plurality of videos without using a hardware internal clock.

The time synchronization device 10 may be implemented in a personal computer (PC), a data server, or a portable device.

Portable devices include laptop computers, mobile phones, smart phones, tablet PCs, mobile internet devices (MIDs), personal digital assistants (PDAs), enterprise digital assistants (EDAs) A digital still camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device or a portable navigation device (PND), a handheld game console, an e-book e-book, or a smart device. A smart device can be implemented as a smart watch, a smart band, or a smart ring.

The sound information may include concepts such as spectrogram, audio fingerprinting, and sound waveform. Audio fingerprinting can be used to locate a desired song in a large music database with a portion of the melody. However, audio fingerprinting can only be used in a no-noise situation.

In general, since there is noise in moving pictures taken at multiple angles, audio fingerprinting may be inappropriate for use in the time synchronization method. In addition, because audio fingerprinting is a concept that is designed to search for music through melody, internal structure may be necessarily modified when used for time synchronization.

The spectrogram can refer to three-dimensional sound information indicating the size of a specific frequency band with time. The spectrogram can express the size of a specific frequency with time in color. For example, green may mean a stronger intensity than blue, and the closer to black the smaller the specific frequency. Thus, since the spectrogram is three-dimensional sound information, it may be difficult to compare spectrograms for time synchronization.

The sound waveform can mean the intensity of the sound over time. Since the sound waveform is easy to understand and two-dimensional information, unlike the spectrogram, it is easy to compare the sound waveforms of a plurality of moving images.

The time synchronization device 10 includes a receiver 100 and a controller 200.

The receiver 100 can receive a plurality of moving pictures. The receiver 100 may include any device including a photographing function. Also, the receiver 100 may receive moving images and images from a device that includes a photographing function.

The receiver 100 may include a plurality of imaging devices, or may include a single device that includes a plurality of cameras. The receiver 100 may output a plurality of received moving images to the controller 200.

The controller 200 can synchronize a plurality of moving images based on the similarity of the plurality of sound waveforms extracted from the plurality of moving images.

Fig. 2 shows a schematic block diagram of the controller shown in Fig.

Referring to FIG. 2, the controller 200 may include a waveform image generator 210 and a synchronizer 230.

The waveform image generator 210 may generate a plurality of waveform images by extracting a plurality of sound waveforms corresponding to each of the plurality of moving images from the plurality of moving images. The waveform image generator 210 may generate a plurality of waveform images from a plurality of sound waveforms using Fast Forward Moving Pictures Expert Group (FFmpeg).

The FFmpeg is a framework for audio, video encoding, and decoding, and the waveform image generator 210 can easily generate a waveform image by extracting a sound waveform from a moving image using FFmpeg.

The waveform image generator 210 may generate a waveform image having a length and an arbitrary height of a multiple of a moving picture frame for precise comparison. The waveform image generator 210 can extract a sound waveform of all the moving images inputted through the FFmpeg and convert the sound waveform into a binary image.

For example, the plurality of waveform images may be a binary image having a length corresponding to twice the frame number of each of the plurality of moving images. In the case of a comparison using a waveform image having twice the length of the frame number, the comparison operation between the two videos can have an accuracy of up to 0.5 frames.

The synchronizer 230 may synchronize the plurality of moving images based on the similarity between the plurality of waveform images.

Figure 3 shows a schematic block diagram of the synchronizer shown in Figure 2;

Referring to FIG. 3, the synchronizer 230 may include a similarity calculator 231 and a time axis adjuster 233.

The similarity calculator 231 may perform a comparison operation between a plurality of waveform images to calculate the similarity.

The similarity calculator 231 can classify the plurality of waveform images into the reference image and the comparison object image based on the lengths of the plurality of waveform images. For example, the similarity calculator 231 can set a waveform image having the shortest length among a plurality of waveform images as a reference image.

The similarity calculator 231 can calculate the similarity with the comparison object based on the reference image. The comparison target image may refer to images other than the reference image among a plurality of waveform images.

The similarity calculator 231 can select a part of the reference image. For example, the similarity calculator 231 may select half of the reference image as part of the reference image.

The similarity calculator 231 may calculate the similarity by performing a comparison operation on a part of the reference image and the comparison image. For example, the similarity calculator 231 may select a part of the comparison object image and perform an XOR (Exclusive-OR) operation between a part of the reference image and a part of the comparison object to calculate the similarity.

The comparison operation may include a logic operation other than XOR. For example, the comparison operation may include AND, OR, NOT, NOR (Not OR), and NAND (Not AND).

The time axis adjuster 233 can synchronize the time axes of a plurality of moving images based on the similarity. For example, the time axis adjuster 233 can coincide the time axis with the point of highest similarity.

4 shows an example of the operation of the time synchronization apparatus shown in Figs.

Referring to FIG. 4, the receiver 100 may receive a plurality of moving pictures. The receiver 100 can photograph directly, including a photographing device, and can receive moving pictures from an external photographing device. The receiver 100 may output a plurality of received moving images to the waveform image generator 210.

The waveform image generator 210 may extract a plurality of sound waveforms from a plurality of moving images and generate a plurality of waveform images based on the extracted sound waveforms. The waveform image generator 210 may output a plurality of waveform images to the similarity calculator 231. [

The similarity calculator 231 may set a reference image among a plurality of waveform images and calculate a similarity by performing a comparison operation on a part of the reference image and a part of the comparison image. The similarity calculator 233 can output the calculated similarity to the time axis adjuster 233.

The similarity calculator 233 can synchronize the time axis by adjusting the time axes of a plurality of moving images based on the similarity. For example, the similarity calculator 233 can adjust the time axis on a frame-by-frame basis.

5 shows an example of an operation in which a synchronizer performs synchronization.

Referring to FIG. 5, the similarity calculator 231 may set an image having a short length among a plurality of waveform images as a reference image, and select a part of the reference image as a part of the reference image. 5, the similarity calculator 231 may select half of the center of the reference image as a portion of the reference image.

The similarity calculator 231 may select a part of the comparison target image and perform a comparison operation with a part of the reference image. For example, a portion of the image to be compared may have the same length as a portion of the reference image.

When the similarity calculator 231 performs a comparison operation, it can be assumed that a plurality of moving images are mostly overlapped.

When the similarity calculator 231 uses the XOR operation as the comparison operation, the larger the difference between the two waveform images, the greater the number of white pixels of the calculation result. That is, it can mean a point having the most similar sound waveform when the number of white pixels is the smallest.

The similarity calculator 231 performs an XOR operation while shifting a part of the comparison object pixel by pixel to find the index when the white pixel is the smallest. For example, the similarity calculator 231 may perform an XOR operation while moving a part of the comparison target image by one pixel from the left to the right of the comparison target image.

The similarity calculator 231 can be assumed to be in the same time zone when the waveform has the closest similarity, that is, when the white pixels are the smallest.

The similarity calculator 231 can calculate the similarity between three or more moving images. In this case, one-to-one comparison operations can be performed on other comparison images based on the reference image.

The similarity calculator 231 can output the similarity calculated by the time axis adjuster 233.

The time axis adjuster 233 can calculate the difference in the number of frames between the two videos by comparing the pixel position of the reference image with the pixel position of the comparison target image. The time axis adjuster 233 can synchronize a plurality of moving images by adjusting the time axis of the moving image by the difference in the number of frames.

FIG. 6A shows an example of a time base synchronization result based on a sound waveform, and FIG. 6B shows an example of a time base synchronization result based on image information.

Referring to FIGS. 6A and 6B, it can be seen that FIG. 6A using sound waveform based synchronization is higher in synchronization accuracy than FIG. 6B using image based synchronization.

For the experiment, three video test sets of 60 frames per second taken on a smartphone can be used. Based on the same test set, the temporal synchronization performance based on image information and the temporal synchronization performance based on sound waveform can be compared. The frame difference after synchronization is shown in Table 1.

Video information based synchronization (frame) Sound waveform based synchronization (frames) Test set 1 7 3 Test Set 2 15 5 Test Set 3 50 3

Referring to Table 1, it can be seen that the sound waveform based synchronization performance is improved as compared with the image information based synchronization method.

Table 2 compares the time taken for time-base synchronization processing between the video information based synchronization method and the sound waveform based synchronization method.

Video information based synchronization (seconds) Sound waveform based synchronization (seconds) Improvement (%) Test set 1 884 One 0.11 Test Set 2 1334 One 0.07 Test Set 3 2138 One 0.05

Referring to Table 2, it can be seen that the time required for the synchronization processing of the sound waveform based synchronization method is significantly shorter than that of the image information based synchronization method.

In case of using image information, the feature should be extracted from all frames. However, in case of using a sound waveform, the complexity of calculation may be lowered because a comparison is performed using a binary image after a sound waveform is formed using FFmpeg. Referring to Table 2, it can be seen that the calculation amount is 0.07% on average, and the calculation amount is decreased.

7 shows a flowchart of a time synchronization method according to one embodiment.

Referring to FIG. 7, the receiver 100 may receive a plurality of moving pictures. The waveform image generator 210 may extract a plurality of sound waveforms from the plurality of moving images to generate a plurality of waveform images (S710).

The similarity calculator 231 can find the shortest waveform image among the plurality of waveform images and set the reference image as a reference image (S720).

The similarity calculator 231 may perform an XOR operation between a part of the reference image and a part of the comparison object (S730). The similarity calculator 231 may store the result of the XOR operation as the number of white pixels (S740).

The similarity calculator 231 can search for an index that minimizes the number of white pixels and select the index with the highest degree of similarity. The time axis adjuster 233 can perform synchronization on the time axis with the highest degree of similarity (S750).

The time axis adjuster 233 may determine whether there remains an image to be compared with the reference image (S760). The time axis adjuster 233 may cause the similarity calculator 231 to perform an XOR comparison operation when there is an image to be compared. The time axis adjuster 233 may terminate synchronization if there is no comparison target.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (19)

Receiving a plurality of moving pictures;
Extracting a plurality of sound waveforms corresponding to each of the plurality of moving images from the plurality of moving images to generate a plurality of waveform images; And
Synchronizing the plurality of moving images based on the similarity between the plurality of waveform images
Lt; / RTI >
Wherein the similarity is calculated by a comparison operation of images generated based on the length of the plurality of waveform images
Time synchronization method.
The method according to claim 1,
Wherein the plurality of waveform images comprise:
A binary image having a length corresponding to twice the frame number of each of the plurality of moving images
Time synchronization method.
The method according to claim 1,
Wherein the generating comprises:
Generating the plurality of waveform images from the plurality of sound waveforms using Fast Forward Moving Pictures Expert Group (FFmpeg)
/ RTI >
The method according to claim 1,
Wherein the synchronizing comprises:
Performing a comparison operation between the plurality of waveform images to calculate the degree of similarity; And
And synchronizing a time axis of the plurality of moving images based on the similarity
/ RTI >
5. The method of claim 4,
Wherein synchronizing the time axis comprises:
Matching the time axis with a time point at which the degree of similarity is highest;
/ RTI >
5. The method of claim 4,
Wherein the calculating step comprises:
Classifying the plurality of waveform images into a reference image and a comparison image based on the lengths of the plurality of waveform images;
Selecting a portion of the reference image; And
Calculating a degree of similarity by performing a comparison operation on a part of the reference image and the comparison object image
/ RTI >
The method according to claim 6,
Wherein said classifying comprises:
Setting a waveform image having the shortest length among the plurality of waveform images as the reference image
/ RTI >
The method according to claim 6,
Wherein the selecting comprises:
Selecting half of the reference image as part of the reference image
/ RTI >
The method according to claim 6,
Wherein the step of calculating the degree of similarity comprises:
Selecting a portion of the comparison image; And
Performing an XOR (Exclusive-OR) operation between a part of the reference image and a part of the comparison object to calculate the degree of similarity
/ RTI >
A receiver for receiving a plurality of moving pictures; And
A controller for synchronizing the plurality of moving images based on the similarity of the plurality of sound waveforms extracted from the plurality of moving images,
Lt; / RTI >
Wherein the similarity degree is calculated by a comparison operation of images generated based on the lengths of the plurality of waveform images generated by extracting the plurality of sound waveforms
Time synchronization device.
11. The method of claim 10,
The controller comprising:
A waveform image generator for extracting a plurality of sound waveforms corresponding to each of the plurality of moving images from the plurality of moving images to generate a plurality of waveform images; And
And a synchronizer for synchronizing the plurality of moving images based on the similarity between the plurality of waveform images
/ RTI >
12. The method of claim 11,
Wherein the plurality of waveform images comprise:
A binary image having a length corresponding to twice the frame number of each of the plurality of moving images
Time synchronization device.
12. The method of claim 11,
Wherein the waveform image generator comprises:
And generating the plurality of waveform images from the plurality of sound waveforms using Fast Forward Moving Pictures Expert Group (FFmpeg)
Time synchronization device.
12. The method of claim 11,
Wherein the synchronizer comprises:
A similarity calculator for performing a comparison operation between the plurality of waveform images to calculate the similarity; And
And a time axis adjuster for synchronizing the time axis of the plurality of moving pictures based on the similarity,
/ RTI >
15. The method of claim 14,
The time-
The time axis is matched to the time point at which the degree of similarity is highest
Time synchronization device.
15. The method of claim 14,
Wherein the similarity calculator comprises:
A plurality of waveform images are classified into a reference image and a comparison object based on the lengths of the plurality of waveform images, a part of the reference image is selected, and a comparison operation is performed on a part of the reference image and the comparison object To calculate the similarity
Time synchronization device.
17. The method of claim 16,
Wherein the similarity calculator comprises:
Setting a waveform image having the shortest length among the plurality of waveform images as the reference image
Time synchronization device.
17. The method of claim 16,
Wherein the similarity calculator comprises:
Selecting half of the reference image as part of the reference image
Time synchronization device.
17. The method of claim 16,
Wherein the similarity calculator comprises:
A part of the comparison object image is selected and an XOR (Exclusive-OR) operation is performed between a part of the reference image and a part of the comparison object to calculate the degree of similarity
Time synchronization device.

Images