KR20140115536A - Apparatus for editing of multimedia contents and method thereof - Google Patents

Abstract

The present invention relates to a device and a method for editing multimedia content which can synchronize voice data and text data during production of multimedia content. The device for editing multimedia content can automatically synchronize voice data and text data through a text object generation unit to generate a text object of a word unit after sequentially separating input text data into paragraph/sentence/and word units; a voice recognition unit to recognize a voice after designating the sentence end position of input voice data and detecting a phoneme section; a voice object generation unit to generate a voice text object from the voice data recognized by the voice recognition unit; and an automatic synchronization unit to synchronize a voice and a text by comparing the text object and the voice text object in a template matching manner, thereby reducing a synchronization operation time compared to an existing manual operation and improving efficiency and accuracy of a synchronization operation.

Description

[0001] Apparatus for editing multimedia contents and method [0002]

The present invention relates to multimedia content editing, and more particularly, to a multimedia content editing apparatus and method for synchronizing voice data and text data in the production of multimedia contents.

Generally, it is necessary to synchronize a voice object and a text object when producing multimedia contents. In particular, synchronizing voice and text in education multimedia content or karaoke household services can enhance educational efficiency or help singers to beat. Here, the voice object (or voice text object) means voice data, and the text object means text data. In addition, the synchronization means matching voice and text.

A general method for synchronizing voice data with text data in the production of multimedia contents is as follows.

A step of visually displaying the text data and the voice data in a time series, a step of selecting a section of the voice data to hear the voice, a step of selecting the same text as the voice to be heard, The text data and the voice data are synchronized through the step of storing the start time and end time information. That is, the synchronization operator repeatedly performs a process of selecting a section to be listened to based on the silent section of the voice data, and mapping the selected section to the corresponding text data after listening.

On the other hand, a conventional technique of synchronizing text data and voice data is disclosed in Published Unexamined Patent Publication No. 1995-0030128 (published on November 24, 1995).

The prior art disclosed in the present invention allows various musical information to be visually transmitted through a monitor of a karaoke system in accordance with a change in the presentation of subtitles and provides musical information visually through morphological changes of song lyrics.

Korean Public Release No. 1995-0030128 (published Nov. 24, 1995)

However, the conventional method of synchronizing the voice object and the text object has a disadvantage in that the synchronization time is required for the time required for the operator to listen to all the voice data.

In addition, the general method as described above has a problem that the accuracy of synchronization varies according to skill of a worker, and the frequency of occurrence of synchronization errors increases due to external factors such as a work situation or mood state of a worker.

In addition, there is a problem that it is impossible to synchronize voice data and text data.

SUMMARY OF THE INVENTION An object of the present invention is to provide a multimedia content editing apparatus and method for synchronizing voice data and text data in the production of multimedia content.

It is another object of the present invention to provide a multimedia content editing apparatus and method which can automatically synchronize voice data and text data in the production of multimedia contents, thereby reducing time required for synchronization and increasing work efficiency.

Yet another object of the present invention is to provide a multimedia content editing apparatus and method for performing synchronization by performing speech recognition on text data and converting recognized results into text to compare the text data and the recognized result text, And a method thereof.

According to an aspect of the present invention, there is provided a multimedia content editing apparatus including: a text object generation unit for sequentially generating input text data in a unit of a paragraph / sentence / word unit and then generating a text object in word units; A speech recognition unit for designating a sentence end position of the input speech data and detecting speech after a phoneme segment is detected; A voice object generation unit for generating a voice text object from the voice data recognized by the voice recognition unit; And an automatic synchronization unit for synchronizing the voice and the text by comparing the text object and the voice text object in a template matching manner.

The multimedia content editing apparatus may further include an asynchronous information generating unit connected to the automatic synchronizing unit to generate a non-synchronized text object as asynchronous information; And an asynchronous information display unit for visually displaying the asynchronous information generated by the asynchronous information generating unit so that the user can manually synchronize the asynchronous information.

According to another aspect of the present invention, there is provided a method of editing multimedia content, the method comprising: (a) generating a text object in units of words from input text data; (b) generating a speech text object through speech recognition from the input speech data; And (c) generating a template of the text object and the speech text object, and performing automatic synchronization by template matching.

According to another aspect of the present invention, there is provided a method of editing multimedia content, comprising: (d) creating a text object template for objects not synchronized in step (c), generating and displaying asynchronization information based on the text object template step; (e) storing attributes of the synchronized object.

The step (a) includes the steps of: (a-1) separating the input text data into paragraphs; (a-2) separating each separated paragraph into sentences; (a-3) separating each separated sentence into words, and generating text data of the separated word units as a text object.

In the step (b), (b-1) designating the end position of the sentence in the inputted voice data; (b-2) automatically detecting a phoneme section based on a silence section in the sentence; (b-3) obtaining speech-recognized text by performing speech recognition with reference to text object information; (b-4) generating the obtained text as a voice text object.

The step (c) includes the steps of: (c-1) generating a set of text templates composed of the text objects in units of words; (c-2) generating a set of speech text templates including a word set in a speech text object formed as a result of speech recognition; (c-3) matching the set of text templates with the set of speech text templates; (c-4) detecting the same word from the template matching result; (c-5) generating the same words detected as synchronization information.

The step (d) comprises: (d-1) constructing a text object template from the unsynchronized text objects in the step (c); (d-2) generating attribute information for each object included in the text object template; (d-3) displaying the non-synchronized period in color on the voice signal display screen; (d-4) generating candidates for dividing a plurality of speech word objects in the section and displaying the candidate points as asynchronous information.

In step (d-2), a template set composed of text objects that are not synchronized automatically is excluded from the text object template set, and a unique sequential number is assigned to the objects of the template set And attribute information about the object immediately adjacent to the object is specified by generating the attribute information.

According to the present invention, it is possible to automatically synchronize voice data and text data, thereby shortening the synchronizing operation time compared to the conventional manual operation.

In addition, according to the present invention, there is an effect of minimizing the occurrence frequency of synchronization errors due to external factors such as the skill level of an operator, a work situation, and a mood state of an operator by automatic synchronization of voice data and text data.

In addition, according to the present invention, efficiency and accuracy of a synchronization operation can be improved by automatic synchronization.

1 is a block diagram of a multimedia content editing apparatus according to a preferred embodiment of the present invention;
2 is a diagram for explaining a silent section for separating voice data in the present invention;
3 is a flowchart illustrating a method of editing multimedia content according to a preferred embodiment of the present invention.
FIG. 4 is a flowchart of an embodiment of the text object generating step of FIG. 3; FIG.
FIG. 5 is a flowchart of an embodiment of a speech text object generation step of FIG. 3;
FIG. 6 is a first example of a speech word divided into six objects in the present invention; FIG.
FIG. 7 is a second example of speech words separated into six objects in the present invention; FIG.
FIG. 8 is a third example of speech words separated by six objects in the present invention. FIG.
FIG. 9 is a fourth example of a speech word divided into six objects in the present invention; FIG.
FIG. 10 is an explanatory diagram for automatically separating a candidate section by a GUI method in the present invention. FIG.
11 is a flowchart showing an embodiment of the automatic synchronization step of FIG. 3,
FIG. 12 is a flowchart of an embodiment of the step of generating and displaying the asynchronous information in FIG. 3; FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A multimedia content editing apparatus and method according to a preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

1 is a block diagram of a multimedia content editing apparatus according to a preferred embodiment of the present invention.

The multimedia content editing apparatus according to the preferred embodiment of the present invention includes a text input unit 10, a text object generating unit 20, a voice input unit 30, a voice recognition unit 40, a voice object generating unit 50, (60), an asynchronous information generator (70), and an asynchronous information display (80).

The text input unit 10 receives text data and the text object generation unit 20 sequentially separates the text data input through the text object generation unit 20 in the order of paragraph / sentence / word It is responsible for creating a word-based text object.

The voice input unit 30 receives the voice data. The voice recognition unit 40 designates the end position of the voice data input through the voice input unit 30, detects the phoneme duration, .

The voice object generation unit 50 generates a voice text object from the voice data recognized by the voice recognition unit 40. The automatic synchronization unit 60 uses the template matching method for the text object and the voice text object It synchronizes voice and text in contrast.

The asynchronous information generating unit 70 generates a non-synchronized text object that is not synchronized with the automatic synchronizing unit 60, and the asynchronous information displaying unit 80 generates the asynchronous information And displays the non-synchronization information generated by the unit 70 visually so that the user can manually synchronize the non-synchronization information.

FIG. 3 is a flowchart illustrating a method for editing multimedia contents according to a preferred embodiment of the present invention, wherein S represents a step.

A method of editing multimedia content according to a preferred embodiment of the present invention includes the steps of: (a) creating a text object on a word-by-word basis from input text data; (b) generating (S20) a speech text object through speech recognition from the input speech data; And (c) generating a template of the text object and the speech text object, and performing automatic synchronization by template matching (S30); (d) generating a text object template for objects not synchronized in step (c), and generating and displaying asynchronization information based on the text object template (S40); (e) storing the attributes of the synchronized object (S50).

In step (c), the object synchronized with the voice data and the text data is moved to step (e) without passing through step (d).

In the step (a), as shown in FIG. 4, (a-1) separating the input text data into paragraphs (S11 to S12); (a-2) separating each separated paragraph into sentences (S13); (a-3) separating each separated sentence into words, and generating separated text data in units of words as a text object (S14).

5, step (b) includes steps (S21 to S22) of designating an end position of a sentence in the input voice data (b-1); (b-2) automatically detecting a phoneme section based on the silence section in the sentence (S23); (b-3) obtaining speech-recognized text by performing speech recognition with reference to text object information (S24); (b-4) generating the obtained text as a voice text object (S25).

In the step (c), as shown in FIG. 11, (c-1) creating (S31) a text template set composed of the text objects of the word unit; (c-2) creating (S32) a set of speech text templates comprising a word set in the speech text object resulting from speech recognition; (c-3) matching the set of text templates with the set of speech text templates (S33); (c-4) detecting the same word from the template matching result (S34); (c-5) generating (S35) the detected identical words as synchronization information.

The step (d) may include: (d-1) constructing a text object template with the unsynchronized text objects in step (c) as shown in FIG. 12; (d-2) generating attribute information for each object included in the text object template (S42); (d-3) displaying a non-synchronized section on the voice signal display screen in color (S43); (d-4) generating a candidate point for dividing a plurality of speech word objects in the interval and displaying the candidate points as non-synchronization information (S44).

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A multimedia content editing apparatus and method according to a preferred embodiment of the present invention will now be described in detail with reference to FIGS. 1 to 12.

First, in order to produce multimedia contents by synchronizing text and voice, the text input unit 10 receives the text data, and the voice input unit 30 receives the voice.

Here, the text input unit 10 may be a keyboard or a portion where text data extracted from a specific text file is input. In addition, the voice input unit 30 may be a microphone for voice signal input or voice data extracted from a specific voice file.

The text data input through the text input unit 10 is transmitted to the text object generation unit 20 and the text object generation unit 20 generates a text object for synchronization from the inputted text data (S10 to S20) .

For example, as shown in FIG. 4, the text object separates the text data input in steps S11 to S12 on a paragraph-by-paragraph basis. Separation of a paragraph unit from input text data is performed by searching for a period from the start text, and if the next data is not searched for a preset data interval to identify the paragraph after the period, it is judged as a paragraph.

Next, each paragraph separated in step S13 is separated by sentence unit. Separation of sentences in paragraphs is based on a period. For example, if the first text in a paragraph is detected and the first time a period is detected, then it is separated by sentence. Then, if the first text is detected after the period, and then the period is detected again after the beginning, this is also separated by sentence.

Finally, in step S14, the separated sentences are separated by word units, and text data of the separated word units is generated as a text object. Here, the text data is separated from the sentence by the word, and if the text is continued but the text is not detected, the text immediately before the point of time when the text is not detected is separated into words. Then, the text data of the separated word unit is generated as a text object. In other words, the text object means a separate word unit. Here, word unit is expressed because the separated unit may be a word or it may be a word including an investigation or the like.

Through this process, the text object generated from the text data is transmitted to the automatic synchronization unit 60.

Meanwhile, the voice recognition unit 40 generates a voice text object from the voice data input through the voice input unit 30 (S20).

As shown in FIG. 5, the speech text object generation is performed by designating the end position of the sentence in the speech data input in steps S21 and S22, and in step S23, the speech section is automatically . Here, as shown in FIG. 2, there is a silent section (silent section) which is a section in which a voice signal exists and a section in which there is no voice signal in the middle of voice and voice. Therefore, the phoneme interval can be automatically detected based on the silence interval.

Thereafter, in step S24, speech recognition is performed by referring to the text object information, and the speech-recognized text is obtained. Here, the speech recognition can use a conventional dynamic time warping (DTW), a hidden markov model (HMM), a continuous speech recognition method using a distributed neural network, and a proposal word speech recognition method.

Next, in step S25, the voice object generation unit 50 generates a voice text object, which is obtained by voice recognition in the voice recognition unit 40, as a voice text object, and transmits the voice text object to the automatic synchronization unit 60. [

The automatic synchronization unit 60 synchronizes the text object with the voice text object in step S30.

Here, the synchronization between the text object and the voice text object is performed by a template matching method. After that, GUI-based synchronization processing for synchronizing text data and voice data, which have not been synchronized in the automatic synchronization process, Is performed.

The synchronization between the text object (word) and the voice object (word) is specified as a synchronization object in which two objects (words) are the same text, that is, they are mapped to each other if they are the same word. Whether or not the same word is judged can be judged by comparing whether the characters expressing the two objects are the same or not. However, a text object can express itself as a character, but a voice object must be re-rendered as a character through another process. The process is enabled using speech recognition techniques. Therefore, whether or not the two objects are synchronized is determined by comparing the character (column) of the text object with the character (column) of the voice object which is a result of recognizing the voice object, and determining whether the character (column) is the same character.

More specifically, as shown in FIG. 11, a text template set is generated based on the text object in units of words input in step S31. Here, the template is a template artificially created for easily comparing the text object and the voice text object with each other.

Since text data must be a reference, a template is created in which a text sentence is a string set of a text object on a word-by-word basis. That is, a set of text templates (A) is generated as A = {aaa, bbb, ccc, ...}.

Next, in step S32, a speech text template set B for a speech word formed as a result of speech recognition is generated. Here, the speech text is a result of converting the speech recognition result into a character string in units of words. The speech text template set B includes B = {a'a'a ', b'b'b', c'c'c ' ...}.

Thereafter, in step S33, a template matching process is performed to match the speech word set B with respect to the template set A as a reference.

The comparison of the strings belonging to the two sets is implemented by sequentially comparing each element (string) of the set B to each element (string) of the template set A 1: 1. That is, the set of template matching elements T is T = {(aaa, a'a'a '), (aaa, b'b'b'), ..., (bbb, b'b'b ' bbb, c'c'c '), ...}, and sequentially compares the elements. If the two object strings match (aaa) = (a'a'a'), It is determined that the object is perfectly matched. For example, if you mark the text template A with the text sentence "Cinderella goes to the palace on a pumpkin carriage," A = {Cinderella is a pumpkin, carriage, ride, going to the palace} It can be seen that this set consists of six word unit objects. If the result of recognizing the speech sentence corresponding to this text sentence is as shown in FIG. 6, then B = {Cinderella, the amber, carriage, ride, goes to the palace) If you are represented as T = {(Cinderella, Cinderella), (pumpkin, pumpkin), (carriage, carriage), (ride, ride), (to palace, to palace) )}, And all the objects are matched, so that the text sentence and the voice sentence are completely synchronized.

Also, if the speech text object of the speech recognition result as shown in Fig. 7 is obtained due to the error of the speech recognition, the two sets A = {Cinderella, pumpkin, carriage, ride, go to the palace}, B = {Cinderella ), T = {(Cinderella, Cinderella), (ride, ride), (to the palace, to the palace), (to go)

We get five matching objects.

At this time, for the unmatched object, the asynchronous information generating unit 70 generates the template A 'as the set of text objects and also generates the unmatched set of the textual objects B'. Here, the set of unmatched objects is expressed as follows.

A '= {Pumpkin, carriage}, B' = {Pumpkin carriage}

In addition, when a speech text object of speech recognition result as shown in FIG. 8 is obtained due to a speech recognition error, a set of non-matched objects is expressed as follows.

A '= {ride a carriage}, B' = {ride a carriage}

In this process, template matching is performed in step S33, the same word is detected in step S34, synchronization attribute information is generated and stored in the internal memory in step S35.

The text data and the audio data that are media synchronization targets have the same characters (columns) originally. That is, the characters (columns) of a text object separated by a word and a voice text object should be completely the same in principle, and the comparison should be judged based on the text object. If there is a complete speech recognition engine, the speech-recognized objects and the text objects will be perfectly matched.

In the present invention, the limited word speech recognition method is applied as an embodiment. In this case, the recognition rate is very high by performing limited recognition only on the words included in the text data. However, there are still cases where the recognition rate of the specific object is extremely low or the recognition is failed because the speech recognition technology is still limited. In addition, when speech data is divided into phonemes according to the silence interval in advance for word-based speech recognition, there is also a mistake due to the separation of the speech words.

For this reason, when the text objects and the speech recognition objects do not coincide with each other, a method of compensating the text objects is necessary. In the present invention, a complementary method by template matching is proposed.

For example, in step S40, asynchronous information is generated and displayed so that only a set that is not matched as described above is synchronized in a GUI manner.

To this end, as shown in FIG. 12, a text object template not automatically synchronized is generated in step S41. In step S42, template object attribute information is generated.

For example, in the process of creating the template set A 'that has not been automatically synchronized, the objects that are not automatically synchronized are assigned unique sequential numbers, and the attribute values that distinguish whether the object immediately adjacent to the right is an automatically synchronized object . At this time, if the attribute value is 0, the right object indicates an auto-synchronized object, and in the case of a non-zero number, indicates a unique sequential number of the right object, which means that the right object is also an object that is not automatically synchronized. That is, the template A 'that has not been automatically synchronized has an attribute value including the number N (N = 1, 2, ..., n) of objects that are not synchronized and the interval information C (N). If C (N) = 0, the object on the right side of the non-synchronized object N means the synchronized object. If C (N) = k (k> N) There is an object that is not synchronized, and its unique number is k.

Therefore, in the case of FIG. 7, the number of objects N = 2, A '(1) = {amber}, A' (2) = {carriage} , C (1) = 2, and C (2) = 0. That is, from the relationship of A '(1) = {amber}, A' (2) = {carriage} and B '= {carriage ride} It can be seen that the object must be split into two. The synchronization process may be performed by the GUI method based on the attribute information C (N) of A '.

As in step S43 for the synchronization process by the GUI method, a section that is not automatically synchronized with template matching is displayed in color on the voice signal display screen. By displaying the un-automated area in color, it increases the concentration of attention so that the operator can grasp at a glance.

In step S44, a candidate point p for dividing the speech word object into n words within the corresponding section is generated. For example, in the case of the speech word object {(pumpkin carriage)} of FIG. 7, since it is necessary to divide into two candidate object regions as shown in FIG. 6, one separation candidate point p is generated as shown in FIG. 10, Will be displayed.

With this automatic display of the unsynchronized section and generation of candidate points, the operator clicks the candidate voice object area to hear it, and if the text is the same as the result of listening, the text object area is clicked with the mouse. When the text object area is selected with the mouse, synchronization between the speech candidate object and the clicked text object is made.

The synchronization information thus synchronized is stored in the internal memory together with the automatic synchronization information. When the multimedia content produced by the synchronization information is reproduced, synchronization between voice data and text data is performed.

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

The present invention is applied to a technique of automatically creating multimedia contents by automatically synchronizing text data and voice data. In particular, it can be effectively applied to the production of educational multimedia contents.

10 ... Text input unit
20 ... The text object creation unit
30 ... Voice input unit
40 ... The voice recognition unit
50 ... The voice-
60 ... Automatic synchronization unit
70 ... The asynchronous information generating unit
80 ... Asynchronous information display

Claims (9)

A text object generation unit that sequentially separates input text data in order of paragraph / sentence / word and then generates a text object in word units;
A speech recognition unit for designating a sentence end position of the input speech data and detecting speech after a phoneme segment is detected;
A voice object generation unit for generating a voice text object from the voice data recognized by the voice recognition unit;
And an automatic synchronization unit for synchronizing the voice and the text by comparing the text object and the voice text object by a template matching method.
The apparatus of claim 1, further comprising: an asynchronization information generation unit connected to the automatic synchronization unit to generate a non-synchronized text object as non-synchronization information; Further comprising an asynchronous information display unit for visually displaying the asynchronous information generated by the asynchronous information generating unit so that the user can manually synchronize the asynchronous information.
(a) generating a text object on a word-by-word basis from the text data input by the text object generating unit;
(b) generating a speech text object through speech recognition from the speech data input by the speech object generating unit; And
(c) generating a template of the text object and the voice text object in the automatic synchronization unit, and performing automatic synchronization by template matching.
The method of claim 3, further comprising: (d) generating a text object template for objects not synchronized in step (c), generating and displaying asynchronization information based on the text object template; and (e) storing attributes of the synchronized object.
The method of claim 3 or 4, wherein the step (a) comprises the steps of: (a-1) separating input text data into paragraphs; (a-2) separating each separated paragraph into sentences; (a-3) separating each separated sentence into words, and generating text data of the separated word units as a text object.
The method according to claim 3 or 4, wherein the step (b) comprises the steps of: (b-1) designating an end position of a sentence in the inputted voice data; (b-2) automatically detecting a phoneme section based on a silence section in the sentence; (b-3) obtaining speech-recognized text by performing speech recognition with reference to text object information; (b-4) generating the obtained text as a spoken text object.
The method of claim 3 or 4, wherein the step (c) includes the steps of: (c-1) generating a set of text templates composed of the text objects in units of words; (c-2) generating a set of speech text templates including a word set in a speech text object formed as a result of speech recognition; (c-3) matching the set of text templates with the set of speech text templates; (c-4) detecting the same word from the template matching result; (c-5) generating the same words as the detected synchronization information.
[6] The method of claim 4, wherein the step (d) includes: (d-1) constructing a text object template from the unsynchronized text objects in the step (c) (d-2) generating attribute information for each object included in the text object template; (d-3) displaying the non-synchronized period in color on the voice signal display screen; (d-4) generating a candidate point for dividing a plurality of speech word objects in the section and displaying the candidate points as non-synchronization information.
[8] The method of claim 8, wherein the step (d-2) comprises the steps of: generating a template set composed of text objects that are not synchronized with each other except for automatically synchronized objects in the set of text object templates; Wherein the attribute information is generated by assigning a sequential number and designating an attribute value of an object immediately adjacent to the object.

Images