KR101996551B1 - Apparatus and method for generating subtitles using speech recognition and script - Google Patents

Abstract

The present invention relates to a system for generating a subtitle file corresponding to a video by using a voice recognition technology under a premise that there is a dialogue material corresponding to the video, wherein the system comprises: an Input unit for inputting video and dialogue information; a preprocessing unit for converting the dialogue information into a list of sentences divided into display sentence units; a word extracting unit which obtains word information including a string of voice by performing voice recognition on the video; a word matching unit for searching and matching the display sentence units including the word information in the list of sentences; a time setting unit for setting a display time in the display sentence units matching the word information; and a subtitle generation unit for outputting, by converting into an electronic file, the list of sentences in which display time is set for the display sentence units.

Description

[0001] APPARATUS AND METHOD FOR GENERATING SUBTITLES USING SPEECH RECOGNITION AND SCRIPT [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a system and method for generating subtitles using voice recognition and a scenario, and more particularly, to a system for generating a subtitles file corresponding to a moving picture Technology.

Subtitles of video have been provided for people who are deaf or not understand foreign language. In order to produce subtitles of a moving picture, it is necessary to take time for a person to find a subtitle piece corresponding to the position where the moving picture is being played and to set the time.

Since the subtitles are made by many individuals, drama, movie, and animated video existing on the Internet may have prologue, intermediate advertisement, or various versions depending on presence or absence of a specific scene and corresponding subtitles. While there are various versions, it is hard to expect that any downloaded subtitles will be compatible with the video. Most of the video players that support subtitles also have the ability to manually adjust the subtitle time, because the subtitles can be a problem in understanding or appreciating the video even if the subtitles are only 2-3 seconds faster or slower. There is also research that automatically adjusts for incompatible subtitle files without having to manually adjust the time.

There is a system that automatically creates subtitles using the speech recognition technology regardless of the existing subtitles. However, there are some aspects that are difficult to use universally because of the limit of the accuracy of the speech recognition technology. Even if the speech recognition technology is correct, unnecessary repetition or grammatical errors appearing in the profanity, abbreviation, and voice may need to be corrected as needed. Thus, the method of writing the speech recognition result in the caption is inherently limited.

In addition, the biggest disadvantage of the automatic captioning function that uses the result of speech recognition as a caption as it is is the accuracy of speech recognition technology. The method of making subtitles by speech recognition is suitable for lecture with less noise to speech or announcement video. However, even in this case, the frequency of errors may differ depending on the pronunciation or intonation of the speaker.

type Contents Abbreviation / Proper Noun Subtitle And it's not just Linux Speech recognition And it's not just lennox Text representation Subtitle I was 21 at the time Speech recognition I was twenty one at the time Caption correction Subtitle I was young Speech recognition I was I young

Table 1 shows the difference between the subtitle and the script that can be obtained by speech recognition. In the case of acronyms and proper nouns, textual representations, and caption correction, it is not an error in a general speech recognition model, which shows that there is a limit to completely substituting subtitles only by speech recognition results. In addition to stuttering, masking of profanity or correcting utterances with unintentional grammatical errors on the subtitles is not an area to judge in speech recognition.

On the other hand, modern speech recognition technology is easily accessible by using cloud service, and its cost is not high. For example, cloud voice recognition services include IBM BlueMix and GCP (Google Cloud Platform). For both services, the cost of converting the entire voice of a movie into text does not exceed the hourly minimum wage in developed countries.

Korean Patent Publication No. 10-2017-0047547

It is an object of the present invention to provide a system and method for generating a caption file corresponding to a moving picture by utilizing a speech recognition technology under the premise that there is metabolism data corresponding to a moving picture, .

According to an aspect of the present invention, there is provided an apparatus for generating speech and a subtitle using a script, the apparatus comprising: an input unit for inputting moving image and metabolism information; and converting the metabolism information into a divided sentence list A word extracting unit for acquiring word information including a character string of a voice by performing speech recognition on the moving picture; and a searching unit for searching for and matching the display sentence unit including the word information in the sentence list A time setting unit for setting a display time in units of the display sentence matched with the word information, and a subtitle generation unit for outputting the electromagnetic wave anomaly and outputting the sentence list in which the display time is set for the display sentence unit .

Further, the word information may further include an utterance time and an end time of the word, and the time setting unit may set the utterance time of the first word of the display sentence unit and the end time of the end word as the display time have.

The preprocessing unit may divide the script into units of one metabase set and set the unit in the unit of the display sentence. If the subtitle is a predefined subtitle, And dividing the display time unit into display time units.

When the subtitle file is input, the time setting unit may set a new display time using a display time previously defined in the subtitle file, for a display sentence unit in which the first word or the end word is not matched have.

The word matching unit may search for the word information among the incompletely matched display sentence units.

Also, the word matching unit may be configured such that a voice expression word is previously set for a number or a special character included in the character string.

According to an aspect of the present invention, there is provided a method of generating a caption using voice recognition and a script, the method comprising the steps of: (A) inputting input moving image and metabolism information; (B) (C) obtaining word information including a character string of a voice by performing word recognition on the moving image by a word extracting unit; (D) The matching unit searching for and matching the display sentence unit including the word information in the sentence list; (E) setting a display time in the unit of the display sentence in which the time setting unit matches the word information; F) The subtitle generation unit outputs an electromagnetic wave an output of the sentence list in which the display time is set for the display sentence unit.

Further, the speech information obtained in the step (C) further includes a speech utterance time and an end time, and the step (E) includes the steps of: And the display time is set.

In the case of the script, the script is divided into units of a metabolism set and set in units of the display sentence. In the case of the subtitle file, the subtitle file includes the subtitle file, Is divided into a predetermined display time unit and is set in units of the display sentence.

In the step (E), when the caption file is input, a new display time is set using the display time previously defined in the caption file, in which the first or last word is not matched. can do.

In the step (D), the word information is searched for among the incompletely matched display sentence units.

Further, the word matching unit may be configured such that a voice expression word is preset for a number or a special character included in the character string.

The apparatus and method for generating a caption using speech recognition and script according to the present invention have the following effects.

First, since the utterance time and the end time of the metabolism detected by speech recognition are used as the active time and the inactivity time of the caption, a caption displayed at the correct time can be generated.

Second, since a prepared script or a subtitle file is used as a display sentence, it is possible to produce a better level of subtitle compared to an automatic subtitle production system using simple speech recognition.

Brief Description of the Drawings Fig. 1 is a block diagram of a subtitle generating apparatus using speech recognition and transcription according to an embodiment of the present invention; Fig.
Fig. 2 is a diagram showing the difference between the display sentence unit and the speech recognition in the same ambassador; Fig.
3 is a diagram illustrating a process of processing a display sentence unit in which word information is matched.
4 is a flowchart of a method of generating a caption using speech and recognition using speech according to an embodiment of the present invention.

An apparatus and a method for generating a subtitle using speech recognition and a script according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

Also, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

In this description, 'time' is a value at a certain moment, and 'time' is a range value having a start time and an end time.

The conventional subtitle generating apparatus using only speech recognition may have incorrect speech recognition results and may be substantially different from the script and actual metabolism. The embodiment of the present invention has an algorithm for searching for a similar sentence in consideration of the above problems and synthesizing the retrieved sentences to generate a final subtitle file. The search algorithm uses a longest common subsequence (LIS) algorithm as an embodiment.

The apparatus for generating subtitles 100 according to an embodiment of the present invention is a system for inputting moving pictures and dialog information and outputting subtitles based on dialog information. Extracts a voice file from the video file, and obtains a voice recognition result in which the utterance time and the end time are given by word using the voice recognition service. In the case of a script, it is converted into a list of sentences composed of one or more words separated by a metabase set unit. If subtitle files whose display time is incompatible with the video can be converted into a sentence list by subtracting only the display time from subtitles. After obtaining the speech recognition result and the sentence list to be the subtitles, the corresponding sentence is searched for in the speech recognition result for every display sentence unit. The search result in the display sentence unit is the position of the first word in the speech recognition result and the position of the end word, and the search result may be plural. Each search result is converted into a single subtitle fragment composed of the text of the script and the ending time of the ending word of the first word matched. When the search for all the sentence list is completed, the subtitle is generated.

Referring to FIG. 1, a subtitle generating apparatus 100 according to an embodiment of the present invention will be described in more detail. The apparatus for generating subtitles 100 according to an embodiment of the present invention includes an input unit 110 for inputting moving picture and dialog information, a preprocessor 130 for converting the dialog information into a sentence list divided into display sentences, A word matching unit 140 for searching for and matching a display sentence unit including word information in a sentence list, A time setting unit 150 for setting a display time in units of display sentences matched with word information, and a subtitle generating unit 160 for outputting a sentence list in which the display time for the display sentence unit is set to an electromagnetic wave.

The meta information inputted through the input unit 110 includes a prepared script or subtitle file. The preprocessing unit 130 divides a script into units of a dialog set, sets the script in units of display sentences, and divides the subtitles into predefined display time units to set them in units of display sentences.

Further, the word information further includes the utterance time and the end time of the word, and the time setting unit 150 sets the utterance time of the first word in the display sentence unit and the end time of the end word as the display time.

One of the necessary information in speech recognition is the speech utterance. Both IBM's Bluemix and Google Cloud's speech recognition services include an option to tell the word's time on a word-by-word basis. [Table 2] is an example of a speech recognition result in which ignition time and end time information are given on a word-by-word basis.

word Firing time (seconds) End time (seconds) this 13.47 13.61 is 13.61 13.73 such 13.73 13.97 as 13.97 14.12 raising 14.12 14.41 your 14.41 14.6 ... ... ...

Before the word matching unit 140 searches for word information and display sentence units, a preprocessing may be required in some cases. The results of the speech recognition are expressed in a certain expression, whereas the script may be composed of speech recognition results and other expressions depending on the nature of the video contents. This is also likely to have the same effect as matching a sentence with a misspelled word. Nevertheless, since the sentence search itself tolerates errors in the matching process to a certain extent, the subtitles can be obtained even if the quality of the speech recognition result is good and the problem of expression is not generally frequent.

The word matching unit 140 decomposes a sentence unit sentence into words, and searches the word sequence (Sequence) in word information that is a speech recognition result. There exist problems such as the above-mentioned inaccuracies in speech recognition and the way in which the string representation methods are different, so that the common partial sequence is searched. If there is an inconsistent partial sequence in the search process, it should be limited in size to prevent partial sequence searches involving excessive errors. For example, FIG. 2 shows a form that can be displayed when a sentence of 'I was 21 at that time' is matched in a speech recognition result. The '21' in the script is correctly recognized as 'twenty one' in speech recognition, but if it matches the string of words without preprocessing, this part remains as an inconsistent part. If '21' is ignored and the corresponding position of the next 'at' is found, it should be close to the previously matched 'was' position. That is, it is necessary to determine how many words the speech recognition failure words can appear on the speech recognition, thereby preventing excessive matching of unnecessary search results. If you do not set this constraint, you can get a lot of search results due to words that appear frequently in sentences such as be verbs or prepositions. Therefore, the word matching unit 140 sets a voice expression word for a number or a special character included in the character string.

Since the word matching unit 140 searches the whole sentence unit in the entire range of the sentence list searching step of the word information, two or more positions can be searched. If the position of the speech information is high and the error rate is high, Can be searched. In order to select the correct result among the multiple results searched or exclude the wrong searched positions, the search results of other sentences must be calculated in a synthesized manner.

Referring to FIG. 3, the word matching unit 140 divides subtitle calculation into two steps. The first is to sort all the displayed sentence units in ascending order by display time. The display time corresponds to one-to-one correspondence with the speech-recognized word information, so that the word information is arranged in order of appearance. If the longest increment sequence is searched based on the order of the display sentence units, the position of each display sentence unit is increased while eliminating the erroneously searched positions, and only one of them is selected to be ready to be converted into the subtitles. In addition to the condition that the order of sentences increases in the process of finding the longest increase sequence, the playback time between the subtitle units should not overlap. This is equivalent to a condition that the recognized specific word information is not included in two or more display sentence units.

In addition, the word matching unit 140 performs a task of scoring a search result to remove unnecessary search results obtained with a low score. The word matching unit 140 performs a method of giving a score of +1 if there is word information matching the display sentence unit and a score of 0 or -1 if there is a word that does not match.

The scoring method solves the following problem. When speech recognition is performed perfectly for the display sentence unit having N words, when all the sentence list is searched, the search result is proportional to N ^ 2 only in the corresponding section. Also, since each of the N words is searched in the entire range, and a common display sentence unit in which only one of the words coincides is also displayed as a search result, it is possible to solve the risk that the search result may be enlarged.

On the other hand, if the score is accidentally low as a result of the search, the corresponding word information may be missed in the subtitles. This missing word information can be supplemented by locally searching for incompletely matched display sentence units. The matching of words that are not matched is not re-searched for all the display sentence units, but the display sentence unit in which words existing before and after the word are completely matched are excluded. By reducing the search range, you can also lower the level of filtering of search results, allowing you to count more cases. It is also possible to apply the idea of re-executing locally to gradually make subtitles to optimize the execution time, or to perform the search again with different search ranges for suspected error parts.

If the first word or end word of the display sentence unit is inconsistent, the subtitles appear to be delayed from the originally displayed moment or disappear later than the original moment.

The time setting unit 150 solves the time inconsistency problem in at least two ways. First, the time setting unit 150 determines that there is word information that does not belong to any display sentence unit even though the matching ends, the end word of the display sentence unit of the corresponding display timing is not matched, or the first word The ignition time or the ending time of the word information is reflected on the active time or the inactivity time of the incompletely matched display sentence unit. This is based on the assumption that words similar in pronunciation are returned during speech recognition, and matching of the word information is not performed.

this is such a strange thing Edit distance this is such 16 this is such as 13 this is such raising 8 this is such raising your 11

[Table 3] shows that when the word matching unit 140 matches up to " this is such " only when the word " this is such a strange thing " This is an example showing that you can match up to 'this is such as raising' by selecting a matching that minimizes.

Secondly, when a caption file is input to the input unit 110, a new display time is set using the display time previously defined in the caption file for the display sentence unit in which the first word or the end word is not matched and the end time is not set . This method is effective on the assumption that the input subtitle file has no problem in dialogue, but the problem is that the time is delayed due to the presence of a specific scene such as a prologue, an epilogue, or an advertisement. If the display time of the display sentence unit is known in advance because the subtitle file is already held, interpolation for non-speech subtitle which can not be processed by speech recognition becomes possible.

When the display time is set for all display sentence units, the subtitle generation unit 160 generates a subtitle file in which display sentence units are integrated. At this time, the extension of the subtitle file may be an extension widely used conventionally such as * .smi, * .srt, and the like.

Next, a subtitle generation method according to an embodiment of the present invention will be described. Referring to FIG. 4, a subtitle generating method according to an embodiment of the present invention includes a step S110 of inputting moving picture and metabolism information by the input unit 110, a step S110 of causing the preprocessing unit 130 to display the metabolism information (S121, S124, S126) of converting the sentence list into a divided sentence list, a step (S130) of acquiring word information including a character string of a voice by performing word recognition on a moving image by the word extracting unit 120, A step S140 for the word matching unit 140 to search for and match a unit of display sentence including the word information in the sentence list, and a step for setting the display time in units of the display sentence matched with the word information And a step (S160) of outputting a sentence list in which the subtitle generating unit 160 sets the display time for the display sentence unit to an electromagnetic wave.

The speech information obtained in step S130 further includes the speech utterance time and the ending time of the speech. In step S150, the speech time of the first word in the display sentence unit and the end time of the end word are set as the display time .

The pre-processing unit 130 includes a pre-prepared script or a subtitle file, and the step of converting the pre-processing unit 130 into a sentence list that is divided into a display sentence unit is a step of, in the case of a script, (S124). In the case of a subtitle file, the subtitle is divided into a predetermined display time unit and set in units of display sentences (S126).

In step S150, when a caption file is input, a new display time is set using the display time previously defined in the caption file for the unit of the display sentence in which the first word or the end word is not matched.

In operation S140, word information is searched for among the incompletely matched display sentence units.

In addition, the word matching unit 140 is characterized in that a voice expression word is preset for a number or special character included in a character string.

[Experimental Example]

Hereinafter, a method of evaluating subtitles generated through the subtitle generating apparatus 100 according to an embodiment of the present invention will be introduced, and the results of experiments with moving pictures and scripts will be described.

Since each word in the display sentence unit is entirely dependent on the script or subtitle file, the evaluation is whether each display sentence unit has the correct display time. However, there is no known evaluation method for the display time of the display sentence unit. Therefore, a new evaluation method was applied in this experiment.

In this experiment, a human made subtitles with a script and a moving picture, and then compared the time of corresponding subtitles. The Jaccard index was used to compare the subtitles.

[Equation 1]

It is possible to calculate the Jacquard exponent by defining the active time and the inactivity time of the display sentence unit as follows. [Equation 2] represents the display time of the display sentence unit. [Equation 3] represents the time at which the display times of two display sentence units overlap.

Original subtitles Active / inactive time

this is such a strange thing 13.28 / 15.46 2.18 Matching speech recognition Firing time / end time

this is such 13.47 / 13.97 0.5
2.18 0.23 this is such raising 13.47 / 14.41 0.94
2.18 0.43

는

와 같으며, 이는 B의 지속시간이다.If we look at the example in [Table 4], if 'this is such a strange thing' is A and 'this is such' is B, when we see the display time, B is a subset of A

The

, Which is the duration of B.

&Quot; (2) "

&Quot; (3) "

The jacquard exponent between subtitle files can be defined as the average jacquard exponent of all display sentence units. This method does not require the user to perform the manual operation again for the evaluation unless the script is changed, and can collect and utilize the video already having the caption for the evaluation. It is reasonable to use it as a relative criterion for the subtitle and to evaluate and improve the similar sentence search or subtitle calculation method, rather than as an absolute criterion, since the time standard may be different depending on the person making the subtitle. In Table 4, even though the 'this is such as raising' is an appropriate match, the display time is more than twice that of the Jacquard index of 0.43, At the discretion of the creator of the video, there is a possibility that it is different from the actual speaking point in the video.

title Play time Number of sentences Number of words TED Announcement 1
Linus Torvalds: The mind behind Linux 21:34 460 3453 TED Announcement 2
Sara Menker: A global food crisis ... 17:53 315 2258 TED Announcement 3
Euna Lee: What I learned as a prisoner ... 12:03 207 1518 TED Announcement 4
Helen Czerski: Fun home experiments ... 16:01 357 3047 movie
Big Short 2:10:28 2429 15585

We will discuss the performance of the system by experimenting with four TED announcement videos available for captioning and one capturing a movie [Table 5] to delete the time information from the subtitles and then creating subtitles using the subtitles as a script. We measured the time taken to generate the subtitles and the errors in the generated subtitles and compared the obtained subtitles with the obtained subtitles. In addition, we compared the average jacquard exponents of subtitling subtitles and non - interpolated subtitles, and verified the method of measuring the performance change when changing the internal algorithm of the system.

CPU Intel Core i3-6100U @ 2.30GHz Memory 20 GB operating system Ubuntu 16.04 LTS avatar Typescript Speech recognition IBM Bluemix

title Speech recognition Create subtitles TED Announcement 1 16 minutes 42 seconds 4380ms TED Announcement 2 8 minutes 20 seconds 1542ms TED Announcement 3 6 minutes 14 seconds 934ms TED Announcement 4 13 minutes 35 seconds 3728ms movie 75 minutes 10 seconds 92602ms

Speech recognition was performed using the IBM Bluemix service, and subtitle calculations were performed with a type script on a laptop PC [Table 6]. As shown in [Table 7], most of the time is used to perform speech recognition. The execution time may vary depending on how performance optimization through partition reduction is configured. The composition used in the experiment was repeated two times and the method was used to exclude the sentences that were searched for less than two words in the first round.

There are two kinds of errors that can occur in generated subtitles. One is when the active and inactive times of the subtitles are inaccurate, and the other is when the subtitles themselves are not present. In Table 7, the number of subtitles in which the Jacquard index calculation with the caption obtained from the inactivity time is inaccurate due to inaccurate active and inactive time is recorded as failure, and the case where the caption itself is omitted is recorded as a missing. The missing ratio is the ratio of the number of misses to the total number of sentences, and the failure rate is the ratio of the number of jacquard exponents to the number obtained by subtracting the number of missed sentences from the total number of sentences.

&Quot; (4) "

&Quot; (5) "

In the case of omission, it is possible that none of the words in the sentence are recognized as speech recognition, and there are no matching words, or the search results that should not be excluded in the process of filtering the search results are excluded. In the case of failure, the sentence and the word are matched to positions that do not actually match, or the original subtitle is written in a way that does not match the timing of the utterance, or the time information of the speech recognition is incorrect.

title Failure (percentage) Missing (percentage) TED Announcement 1 1 (0.2%) 28 (6.1%) TED Announcement 2 2 (0.6%) 2 (0.6%) TED Announcement 3 0 (0.0%) 8 (3.9%) TED Announcement 4 6 (1.7%) 11 (3.1%) movie 31 (1.6%) 441 (18.2%)

 [Table 8] shows that the movie has a higher omission rate than the TED announcement, which indicates that the voice of the movie is difficult to recognize than the voice of the TED. In case of failure rate, both the TED announcement and the movie are shown as low, which means that the probability that the subtitle is in the wrong position when the subtitle matches at least one word is included in the subtitle.

title Before interpolation After interpolation TED Announcement 1 0.72 0.75 TED Announcement 2 0.59 0.60 TED Announcement 3 0.57 0.62 TED Announcement 4 0.63 0.68 movie 0.46 0.53

 The average of the jacquard exponents of each sentence can be used as a relative system evaluation measure. Execution time [Table 9] shows that the subtitles corrected by interpolation are closer to human-made subtitles.

Average Jacquard Index 0 seconds (original) One 0.25 sec 0.79 0.5 second 0.64 1 second 0.39 1.5 seconds 0.23 2 seconds 0.13 3 seconds 0.04

 Table 10 summarizes the average jacquard exponent with the original by giving a collective error to all caption fragments of the TED announcement 1 video caption. Considering that the subtitle control function of a general video player is adjusted in 0.5 second units, the average Jacquard index should be 0.64 or more so that it is a relatively accurate subtitle.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It is clear that the present invention can be suitably modified and applied in the same manner. Therefore, the above description does not limit the scope of the present invention, which is defined by the limits of the following claims.

100: caption generation device 110: input part
120: word extracting unit 130: preprocessing unit
140: word matching unit 150: time setting unit
160: Subtitle generation unit

Claims (12)

An input unit for inputting video and metabolism information;
A preprocessing unit for converting the metabolism information into a sentence list divided on a display sentence unit basis;
A word extracting unit for performing speech recognition on the moving image to obtain word information including a character string of a voice;
A word matching unit for searching for and matching the display sentence unit including the word information in the sentence list;
A time setting unit for setting a display time in units of the display sentence matched with the word information;
And a subtitle generation unit for outputting the sentence list in which the display time is set for the display sentence unit,
Wherein the word information further includes an utterance time and an end time of the word,
Wherein the time setting unit sets the display time of the first word of the display sentence unit and the end time of the end word as the display time,
Wherein the meta information input through the input unit includes a prepared script or a subtitle file,
Wherein the preprocessing unit divides the script into units of a metabase set and sets the script in a unit of the display sentence, divides the caption into units of a predetermined display time,
Wherein the time setting unit sets a new display time using a display time previously defined in the subtitle file when the subtitle file is inputted, wherein a display sentence unit in which a first word or an end word is not matched is set in the subtitle file, . delete delete delete The method according to claim 1,
Wherein the word matching unit searches the word information among the incompletely matched display sentence units. The method according to claim 1,
Wherein the word matching unit sets a voice expression word for a number or a special character included in the character string. (A) receiving input moving image and metabolism information;
(B) the pre-processing unit converting the metabolism information into a sentence list divided on a display sentence unit basis;
(C) a word extracting unit performing speech recognition on the moving image to obtain word information including a character string of a voice;
(D) a word matching unit searching and matching the display sentence unit including the word information in the sentence list;
(E) setting a display time in the display sentence unit in which the time setting unit matches the word information;
(F) a subtitle generating unit outputting the electromagnetic wave anomaly of the sentence list in which the display time is set for the display sentence unit,
Wherein the word information obtained in the step (C) further includes an utterance time and an end time of the speech,
Wherein the step (E) sets a speaking time of the first word of the display sentence unit and an ending time of the end word as a display time,
The meta information includes a prepared script or subtitle file,
In step (B), the script is divided into a plurality of display sentences by dividing the script into units of one metabase set. In case of the subtitle file, the subtitle is divided into a predetermined display time unit and is set in units of the display sentence ,
Wherein in the step (E), when a subtitle file is input, a new display time is set using a display time previously defined in the subtitle file, Generation method. delete delete delete 8. The method of claim 7, wherein step (D)
And searching the word information among the incompletely matched display sentence units. 8. The method of claim 7,
Wherein the word matching unit sets a voice expression word for a number or a special character included in the character string.

Images