JP2010157816A - Subtitle information generating device, subtitle information generating method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a subtitle information generating device for generating a subtitle group having timing information reduced in deviation from actual speech timing. <P>SOLUTION: When an isochronal clause pending character string being a timing unit character string where the number of isochronal clauses is not uniquely determined exists between the pair of timing unit character strings having original timing information, the subtitle information generating device obtains a differential time between the timing information of the determined timing unit character string and the original timing information, acquires respective kinds of timing information by subtracting the differential time from the original timing information of the pair of timing unit character strings, and determines the timing information of the isochronal clause pending character string existing between by the use of the timing information of the pair of timing unit character strings. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a caption information creation apparatus, caption information creation method, and program for creating caption information.

  In broadcasting or the like, Closed Caption (hereinafter referred to as “CC”) in which the content of audio is converted into a character string is distributed along with the audio. In order to respond to the viewer's request for the display format such as the number of characters and lines of CC displayed at one time, the receiving side creates a set (hereinafter referred to as subtitle group) according to a certain format based on the CC, and each subtitle group A method for providing timing information is disclosed (see Patent Document 1).

  In live broadcasting, there is a difference between the CC display timing and the corresponding voice utterance timing. Furthermore, the display state is different from the CC of the recorded broadcast edited in consideration of readability under sufficient work time, for example, several seconds pass while only one word is displayed on the screen.

  In order to solve this problem, it is necessary to create a caption group on the receiving side and give timing information. Since the partial character string (hereinafter referred to as “timing unit character string”) that may be at the front end of each subtitle group depends on the contents of the CC and the viewer's request, the timing unit character string is set in as small a character string unit as possible. In addition, it is necessary to be able to calculate timing information for all timing unit character strings.

In the invention of Patent Document 1, based on the timing information at which the timing information is determined (hereinafter referred to as “base point position”), the number of characters and the character type, or phoneme existing between the base point position and the front end of the caption group. The timing information at the front end of the caption group was determined by analogy.
JP 2000-350117 A

  In a situation where there is a difference between the CC display and the voice utterance timing, a character string (hereinafter referred to as “mora undecided character string”) in which the number of corresponding mora cannot be determined as one between the base point position and the front end of the caption group. ) May be included.

  In such a case, the invention of Patent Document 1 has a problem that timing information of each timing unit character string located between the mora undecided character string and the front end of the caption group cannot be calculated. In addition, since the size of the error caused by the difference in the determination process of each timing information is not taken into account, there is a problem that the timing information given to the caption group may be largely deviated from the actual utterance timing.

  Accordingly, the present invention has been made to solve the above-described problems, and provides a caption information creating apparatus and method for creating a caption group having timing information with little deviation from the actual speech timing. To do.

  The present invention provides a subtitle character creation group for creating subtitle information for audio, and a reference correspondence between a subtitle character string group created in advance and the audio given for each arbitrary number of characters in the subtitle character string A timing unit character string that receives original subtitle information including original timing information that is time information indicating the timing information that is time information for displaying the subtitle character string in synchronization with the audio is represented by the subtitle character. An extraction unit that extracts from a group of columns, the speech and the timing unit character string are associated by speech recognition, and the timing unit character string that has been associated with the timing unit based on the time information of the speech A first timing information determination unit for determining information, and the timing timing character string for which the timing information has been determined. The timing of the undecided timing unit character string for which the timing information could not be determined using the number of isochronous segments that are isochronous segments included in each timing unit character string with reference to the timing information An isochronous segment that is a timing unit character string in which the number of isochronous segments cannot be uniquely determined between a second timing information determining unit that determines information and a pair of the timing unit character strings having the original timing information When there is an undecided character string, (1) a difference time between the timing information of the determined timing unit character string and the original timing information is obtained, and (2) from the original timing information of the pair of timing unit character strings. The timing information is obtained by subtracting the difference time, and (3) a pair of timing unit sentences A third timing information determining unit that determines the timing information of the isochronous segment undecided character string in between using the timing information of the column, and a subtitle character obtained by dividing the subtitle character string group according to an arbitrary format A subtitle group that is a column and each of the determined timing unit character string, the undecided timing unit character string, or the isochronous segment undecided character string for displaying each subtitle group in synchronization with the audio A subtitle information generation apparatus comprising: a generation unit that generates the subtitle information including timing information.

  According to the present invention, it is possible to create a caption group having timing information with little deviation from the actual speech timing.

  Hereinafter, a caption information creating apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  In each of the following examples, when applying the caption information creation device to Japanese, “Mora” is used as the segmental unit of isochronous sound, and when applying to English, a set of fixed syllables is applied. Use the foot that is. In other words, “isochronous sound segment unit” refers to mora, foot, and the like. Mola and foot will be explained later.

  Hereinafter, a caption information creating apparatus 10 according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 6 and FIG. 10.

(1) Configuration of Subtitle Information Creation Device 10 FIG. 1 is a block diagram showing a caption information creation device 10 according to the present embodiment.

  This subtitle information creation device 10 includes a timing unit character string extraction unit 100, a voice base timing information determination unit 101, a mora number determination unit 102, a mora base timing information determination unit 103, a difference time base timing information determination unit 104, and a subtitle group generation. Part 105.

  Note that the caption information creating device 10 can also be realized by using, for example, a general-purpose computer device 10 as basic hardware. That is, the timing unit character string extraction unit 100, the voice base timing information determination unit 101, the mora number determination unit 102, the mora base timing information determination unit 103, the difference time base timing information determination unit 104, and the subtitle group creation unit 105 This can be realized by causing a processor mounted on the computer apparatus 10 to execute a program. At this time, the caption information creating device 10 may be realized by installing the above program in the computer device 10 in advance, or may be stored in a storage medium such as a CD-ROM or via the network. May be distributed and this program installed in the computer apparatus 10 as appropriate.

  The timing unit character string extraction unit 100 extracts a timing unit character string from the caption character string group of the original caption information.

  The voice base timing information determination unit 101 associates the timing unit character string with the voice, and determines timing information of each timing unit character string based on the result.

  The mora number determination unit 102 determines the mora number of each timing unit character string.

  The mora base timing information determination unit 103 determines the timing information of the pending timing unit character string based on the number of mora of each timing unit character string and the timing information of the determined timing unit character string.

  The difference time base timing information determination unit 104 determines the timing information of the determined timing unit character string and the original subtitle when the mora undecided character string exists between the determined timing unit character string and the undecided timing unit character string. Based on the information timing information, the timing information of the pending timing unit character string is determined.

  The caption group creation unit 105 creates a caption group.

(2) Operation of Subtitle Information Creation Device 10 Hereinafter, the operation of the caption information creation device 10 according to the first embodiment will be described with reference to FIG. 2 and other drawings. FIG. 2 is a flowchart of the caption information creating apparatus 10 according to the first embodiment.

  Here, as an example, a description will be given assuming a television having a function of displaying CC as shown in FIG. The audio and the original subtitle information are received as broadcast radio waves, and the audio and original subtitle information extracted from this are input to the subtitle information creating apparatus 10 of the present invention.

(3) Timing Unit Character String Extraction First, in step S0 of FIG. 2, the timing unit character string extraction unit 100 extracts a timing unit character string from a caption character string group included in the original caption information.

  Individual subtitle character strings constituting the subtitle character string group are displayed on the screen at a time in synchronism with audio, and display start and display end timing information respectively as shown in c1 to c5 of FIG. Is attached. In FIG. 3, the display start time, the display end time, and the subtitle character string are indicated by “,” delimiters, respectively. The display start and end times are in seconds with the audio playback start time being 0 seconds.

  The timing unit character string to be extracted is a fixed character pattern. In the caption information creation device 10, the first is the association element on the caption character string side when the speech and the caption character string are associated, and the second is the caption group. It has the role of leading edge candidate. Therefore, it is desirable that the character pattern defined as the timing unit character string is a character string composed of as few characters as possible for association with audio, and is a unit that can be the front end of a caption group. It becomes.

  Here, “word” is taken as the character pattern unit of the timing unit character string, and the above “can be the front end of a caption group” will be described. For example, the word “Olympic” can be the front end of a caption group, as shown below.

Subtitle group 1: “Now, next week”
Subtitle group 2: “The Olympics will begin”.

  Assuming that the “Olympic” is divided into “Ori”, “N”, and “Pick” in more detail, subtitle groups that begin with “N” are appropriate from the viewpoint of readability when displayed. Is hard to say.

Subtitle group 1: “Now, it ’s finally next week”
Subtitle group 2: “The pick begins.”
Therefore, “N” in “Olympic” cannot be the front end of the subtitle group. It is more appropriate to treat the word “Olympic” as a unit.

  The timing unit character string need not be limited to Japanese. For example, a specific English character pattern such as “IN” or “THE” may be set as the timing unit character string. Therefore, the subtitle character string handled in the present invention does not depend on the language. English will be described in detail in a later example.

  For the sake of simplicity, hereinafter, a Japanese word is used as the timing unit character string, and a word obtained from the caption character string is hereinafter referred to as a caption word. Subtitle words are extracted using a known technique such as simple partial character string search or morphological analysis.

(4) Determination of Speech Base Timing Information Next, in step S1 of FIG. 2, the speech base timing information determination unit 101 performs speech recognition from the received speech and extracts a word (hereinafter referred to as “speech word”). . Correlate with subtitle word sequence using dynamic programming. Timing information of a subtitle word (hereinafter referred to as “match word”) in which a corresponding voice word is found is given as timing information of the voice word.

  For example, as shown in FIG. 4, “Olympic” on the speech word string is associated with “Olympic” on the caption word string, and 60.88 (display start time) and 61. 63 (display end time) is given as timing information on the caption word side.

  After the assignment, it is determined in step S2 whether timing information of all the caption words has been determined. If so, a caption group is created in step S6 and the process ends.

  The speech word string in FIG. 4 is a list of speech recognition results and includes recognition errors. In the example, “morning”, “no”, “book”, “da”, and “outside” are speech words with recognition errors.

(5) Determination of Mora Base Timing Information When there is a subtitle word for which timing information is not determined (hereinafter referred to as “mismatched word”) in the determination of step S2 in FIG. Base timing information determination is performed to determine timing information for mismatched words.

  At this time, the mora number determining unit 102 determines the mora number of each word and provides the determined number to the mora base timing information determining unit 103.

  “Mora” refers to a segmental unit of sound having a certain length of time. The length of time varies depending on the language or dialect even within the same language. All languages have syllables, but some languages do not have mora. In Japanese, in principle, one kana is pronounced with the same length (one note), and one note corresponds to one mora. However, the roaring sound is recognized as 1 mora. In addition, the sound repellent “N”, the long sound “-”, and the prompt sound “T” are also 1 mora. Use English syllables instead of mora.

  The determination of mora base timing information will be described with reference to FIG.

  The mora base timing information determination unit 103 calculates the time per mora using the timing information of the matching word and the number of mora of each word, and calculates the timing information of the caption word by analogy based on the time. For example, when determining the timing information for each of the mismatch words “S” and “Japan” between the match words “participation” and “player”, the time per mora is calculated by the following formula.

m = (T2-T1) / mt

Here, m is the time per mora, T1 is the display end time of “participation”, T2 is the display start time of “player”, and mt is the total number of mora between “participation” and “player”.

  Since “do” is 2 mora and “Japan” is 3 mora, mt is 5, and m is 0.156 as a result of (63.38-62.60) / 5.

Based on m, it is determined that m × 2 time elapses for the “to do” utterance and m × 3 time elapses for the “Japan” utterance. Therefore, the timing information for “Yes” and “Japan”

“Yes” display start time = T1
“Yes” display end time = T1 + m × 2
“Japan” display start time = T1 + m × 2
"Japan" display end time = T2

It is obtained by.

  As described above, the mora base timing information determination unit 103 is based on the premise that subtitle words are continuous without being spaced in time series. In other words, it is assumed that there is no vacancy for a certain time or more between the utterance timings of adjacent words.

  Therefore, as shown in FIG. 3, a group of subtitle words that are continuous in time series (hereinafter referred to as “time-series continuous word group”) based on the original timing information obtained from the original subtitle information before the calculation is performed. After that, the timing information is determined by calculating each group. In the original subtitle information, a group separates the subtitle character strings when there is a certain time difference between the display end time of one subtitle character string and the display start time of the next displayed subtitle character string. Create it. For example, when the maximum time difference that is considered to be continuous in time series is determined to be 2.00 seconds, 33.37-30.40 = 2.3 between the display end time of c2 and the display start time of c3. Since there is a time difference of 97 seconds, the range is divided here.

  Now, as described above, the timing information of the mismatch word sandwiched between the two match words is determined.

  Next, determination of timing information of mismatched words that are not sandwiched between matching words will be described.

  If the subtitle words at the front and rear end of the time-series continuous word group are not matching words, the unmatched words existing from the front end of the time-series continuous word group to the first matching word in the group are sandwiched between the above-mentioned “matching words”. Not a mismatched word ". The situation at the rear end is the same except that the front and rear ends are reversed, and only the front end will be described as an example.

In FIG. 5, it is assumed that the word “SAA” is a word corresponding to the front end of the time-series continuous word group. At this time, to determine the two timing information of “Sa” and “Today”, the first matching word “Olympic” as viewed from the front end in the group is used to calculate the time per 1 mora and use it. . From the display start time and end time of the 6-mora “Olympic”, the time per 1 mora is obtained as (61.63-60.88) / 6.

“Sa” display start time = T3-m × 4-m × 2
“Sa” display end time = T3-m × 4
“Today” display start time = T3-m × 4
"Today" display end time = T3
"Olympic" display start time: T3
Time per mora: m

And decide. As the time per 1 mora, the time per 1 mora that has already been calculated at another location in the group may be used. In other words, the time per 1 mora calculated based on two matched words and a mismatched word sandwiched between them may be used. Moreover, the time per 1 mora about each matching word in a group may be calculated, and those averages may be used.

(6) Determination of differential time base timing information There is a word (hereinafter referred to as “mora undecided word”) in which the number of mora cannot be determined by the mora number determination unit 102.

  The first is a numeric character string that does not know whether it is a bar or digit, the second is a symbol whose corresponding mora number is not constant, such as “,”, “.”, “...”, And the third is a mora number determination unit 102 It is a character pattern that is not done. The “non-corresponding character pattern” refers to an unregistered word when a dictionary is used in the mora number determination unit 102, for example.

  In the example of FIG. 6, “3324” is a mora pending word.

  If “3324” is digit reading, “Sanzen San Byakuni Juyong” is 13 mora. If it is a stick reading, “Sansan Nyon” is 8 mora. Further, for example, in the case of a stock market, “Sanzen San Byakufu Juyon” is 14 mora. Thus, the actual utterance for the same notation of CC depends on the context, and further knowledge and calculation amount are required for automatic determination. If the calculation is determined by any one of the reading methods and the reading method is different from the original reading method, a large error may occur. The calculation result of the error caused by the difference between bar reading and digit reading is shown below.

Time per mora based on the matching word “participation” = (64.93−64.48) /3=0.15

Error due to difference between “2224” digit reading and bar reading mora = 0.15 × (13−8) = 0.75

Since there is an increased possibility that the above error will occur each time a mora pending word is mixed, the mora base timing information determination unit 103 alone may not be able to calculate timing information with a substantially small error.

  Therefore, when determining the mora base timing information in step S3, if there is a case where a mora undecided word exists between the matched word and the unmatched word in the time series continuous word group, the difference time base timing is determined in step S4. The information determination unit 104 determines the timing information of the mismatch word.

  The difference time base timing information determination unit 104 uses timing information of original caption information (hereinafter referred to as “original timing information”). This original timing information is given for each subtitle character string in the unit of one-screen display in the original subtitle information, but when it is subtitle information given in real time in live broadcasting, it is completely synchronized with the audio. That's not true. These display start times are regarded as display start times of words corresponding to the front end of each subtitle character string. For example, the display start time of the subtitle character string of c2 in FIG. 3 is 27.48, and this is regarded as the display start time of the front end word “Sa” of the subtitle character string of c2 (e2). Similarly, the words e1 and e3 are words having a display start time (hereinafter referred to as original display start time) based on the original timing information.

  With reference to FIG. 6, the flow of determining the difference time base timing information will be described.

  First, as indicated by a circled number 1 in FIG. 6, timing information of the word “previous” having a display start time based on the original timing information is calculated based on the matching word “participation” by determining the mora base timing information. .

  Next, as indicated by a circled number 2 in FIG. 6, a difference time 7 .7 between the “previous” original display start time 71.38 and the display start time 63.88 of the timing information calculated by determining the mora base timing information. 5 is calculated.

  Next, as indicated by a circled number 3 in FIG. 6, the display start time of 60.88 is obtained by subtracting this difference time from the original display start time 68.38 of “this year”.

  Thereafter, the mora base timing information determination unit 103 calculates timing information from “this year” to “ha”.

  For “to”, the timing information is determined using the mora base timing information determination unit 103 with “participation” as the calculation base point.

  With the above processing, the timing information of the adjacent words in both the front and rear ends is found, and the timing information of the mora pending word “3324” is determined.

(7) Subtitle group creation As shown in FIG. 2, timing information is provided for all words by audio base timing information determination (step S1), mora base timing information determination (step S3), and differential time base timing information determination (step S4). If determined, next, in step S6, the caption group creation unit 105 creates a caption group.

  The creation follows firstly the timing information determined by (S1) to (S4), secondly the user's desire for the subtitle display form, and thirdly the format considering readability.

  If the display end time of a word and the display start time of the next word are not the same in the first determined timing information, the two words are not considered as the same caption group.

  The second user's request for the subtitle display form may be the designation of the number of lines and the number of characters.

  Examples of the format considering the third legibility are as follows.

  ・ Determine the number of characters to be displayed at one time and the display time of the data, taking into account the human reading speed obtained statistically.

  ・ Prevent punctuation marks at the beginning of the line.

  ・ Do not delimit immediately after the prefix or immediately before the suffix.

  The subtitle group may be created by a method as shown in claims 1 and 2 of the invention of Patent Document 1.

  Then, the display start time of each subtitle group front end word is defined as the subtitle group display start time, and the subtitle group rear end word display end time is defined as the subtitle group display end time.

  Regarding how to determine the display start and end times of a caption group, it is conceivable that a time difference is intentionally given based on user requests and readability.

  For example, as shown below, the subtitle utterance time is set longer than the actual utterance time to cope with the case where the assumed subtitle reading speed of the user is slower than the actual utterance.

  ・ If there is a certain time difference Td between the display end time of the subtitle group N and the display start time of the subtitle group displayed next to N, the display end time of the subtitle group N is added to Td or less. Thus, the subtitles are displayed for a certain time after the utterance corresponding to the subtitle group N ends.

  ・ If there is a certain time difference Td between the display start time of subtitle group N and the display end time of subtitle group displayed before N, subtract Td or less from the display start time of subtitle group N Thus, the subtitle group N is displayed from a predetermined time before the utterance corresponding to the subtitle group N starts.

(8) Effect According to the present embodiment, it is possible to calculate the timing information of the timing unit character string in which the mora pending character string exists between the base point position on the CC and the mora pending character string in the mora pending range. Thus, a subtitle group having these character strings as the front end can be created, and the demand for the CC display state can be met.

  That is, the difference time between the timing information of the base point position and the original timing information of CC (timing information given to each subtitle character string for one screen display in CC) is calculated, and using this, the mora undecided character string and thereafter The timing information of the timing unit character string located at can be interpolated.

  Hereinafter, the caption information creating apparatus 10 according to the second embodiment of the present invention will be described with reference to FIGS.

  In this embodiment, the mora number estimation unit 106 will be described. For convenience, an undecided range composed of words will be described as an undecided word range. The undecided word range is a range in which the front and rear ends are mora undecided words, and there is no matching word or word having the original display start time in the range. However, if multiple ranges are adjacent or overlap, only the maximum range that combines them is allowed.

(1) Configuration of Subtitle Information Creation Device 10 FIG. 7 is a block diagram showing the caption information creation device 10 according to the present embodiment. The description will be made only on portions different from the first embodiment.

  The mora number estimation unit 106 determines an undecided word range and estimates the number of mora for each word in the undecided word range.

  The mora base timing information determination unit 103 determines the timing information of each word in the pending word range based on the estimated number of mora.

(2) Estimation of the number of mora The estimation of the number of mora will be described with reference to FIG.

  Three words “200”, “from” and “300” on the subtitle word string constitute one undecided word range. First, the total time elapsed in the pending word range is calculated from the display end time of the word “now” adjacent to the front of the pending word range and the display start time of the word “no” adjacent to the backward. In the example, 74.01-72.66 = 1.35 is obtained.

  Next, all combinations of mora number candidates for each word in the undecided word range are created, and the time per mora is calculated for each combination based on the total time of the range.

  “200” is a digit reading “Nihyaku” (3 mora) or a stick reading “Knee Zero Zero” (6 mora). "Zero zero" (6 mora). The number of combinations is 2 × 1 × 2 = 4. The time per 1 mora is calculated for each combination.

  From each calculated time, one is selected based on the time per 1 mora around the pending word range. In FIG. 8, since the time per mora used for the calculation of the word “no” adjacent to the back of the pending word range candidate is 0.15, 0.15 is calculated as the time per mora. The combination “Nihyaku” (3 mora) + “From” (2 mora) + “Sanjak” (4 mora) is selected and used as the estimation result.

  As the selection method, for example, the time per one mora used for determining the timing information of the words before and after the pending word range may be averaged, and the time closest to the value may be selected.

(3) Effect According to the present embodiment, a range having one or more mora undecided character strings (front and rear ends are mora undecided character strings and do not include a base point position inside. Further, undecided word ranges are adjacent or overlapped. For each of those in the pending word range, the number of mora actually spoken is estimated from the timing information before and after the pending word range and the number of mora numbers in the pending word range. The timing information of the mora pending character string can be determined.

  Hereinafter, the caption information creating apparatus 10 according to the third embodiment of the present invention will be described with reference to FIG.

  This embodiment relates to creation of a caption group using a timing information determination process.

(1) Timing-accompanying information When timing information is determined, information indicating “what process has been determined” is used as timing-accompanying information. There are various processes for determining timing information. For example, the user may input timing information while listening to the received voice.

  Further, as described below, each timing information determination unit described above may be used.

M1: Voice base timing information determination unit 101
M2: Mora base timing information determination unit 103
M3: Difference time base timing information determination unit 104
M4: Mora number estimation unit 106
The mora number estimation is also added as “M4: mora number estimation unit 106”. The reason why the mora number estimation is added as one element of the variation is that it is considered that there is a difference in error in the obtained timing information depending on whether the mora number estimation unit 106 is used or not.

(2) Process Variations The following are variations of the process that timing information determines.

1. M1
2. M1 + M2
3. M1 + M3
4). M1 + M2 + M3
5). M1 + M2 + M4
6). M1 + M3 + M4
7). M1 + M2 + M3 + M4
The time error is the smallest when it is determined only by M1 “determined only by the voice base timing information determination unit 101” in the front row. Thereafter, the lower the level, the greater the possibility that the temporal error in the obtained timing information will increase. Here, the time error is an error between the audio time and the caption display time.

(3) First Situation in which Error Occurs A situation in which a temporal error occurs will be described by taking “M1” and “M1 + M2” as examples.

  When the matched word “player” in FIG. 5 is compared with the unmatched word “Yes”, the timing information of “Yes” assumes that “Enter”, “Yes”, “Japan”, and “Player” are spoken at the same speed. It is said. For this reason, if the speaking speed is different when speaking “Yes” and “Japan”, an error occurs. On the other hand, since the matching word is given timing information based on the association with the voice, the possibility of an error is small compared to “Yes”.

  Further, regarding the “M2: mora base timing information determination unit 103” among the variations, there is a difference in the error of the timing information determined even with the same means. The factors causing the error in the mora base timing information determination unit 103 are listed below. The mismatch word that determines the timing information is referred to as “the word” in the following factor list.

  The first factor is the number of mora between the word and the matching word that is the calculation starting point. The smaller the number of mora, the less likely that an error will occur. This is because the time error per 1 mora increases as the number of mora to be multiplied increases.

  The second factor is whether or not the word is sandwiched between matching words. The possibility that an error will occur is smaller when it is sandwiched than when it is not sandwiched. This is because the time per mora can be calculated based on the timing information of both matching words when sandwiched, but cannot be done when not sandwiched.

  The third factor is the number of matching words in the time-series continuous word group including the word. The larger the number of matching words, the smaller the possibility of errors. This is because as the number of matching words increases, the time per mora can be calculated from each of them, and the possibility of an error in the time per mora is reduced.

(4) Second Situation in which an Error Occurs In the mora base timing information determination unit 103, there is a case where the coincidence word serving as the calculation base point does not exist in the time-series continuous word group G. In this case, using the word having the original display start time included in the time series continuous word group before and after G and the word having the original display start time included in G, the difference time is the same as the difference time base timing determination unit. A method of interpolation is conceivable. The timing information of each word in G for which the timing information has been determined based on the word for which timing information is obtained using this method is the timing information of the words in the time-series continuous word group other than G that does not use this method. In comparison, there is a greater possibility of errors.

(5) Third Situation in which Error Occurs Further, in “M4: mora number estimation unit 106”, the possibility that an error occurs due to the following factors is different.

  There is a time difference between the time per 1 mora calculated based on the combination of the number of mora in the estimation result and the time per 1 mora of the combination candidate not selected in the estimation result. As this time difference is larger, the possibility that the mora number estimation result is incorrect is reduced, and the possibility that the given timing information includes an error is reduced.

  Also, in the estimation of the number of mora, when there is no difference in the time per one mora obtained from the combination and the number of mora number combination candidates cannot be determined as one, the possibility that the mora number estimation result is erroneous is high.

(6) Summary There are many variations in the process of determining timing information as described above.

  The determination process affects the magnitude of the error in the timing information to be determined. The caption information creating apparatus 10 according to the present embodiment records such a determination process as timing-accompanying information and uses it when creating a caption group.

(7) Configuration of Subtitle Information Creation Device 10 FIG. 9 is a block diagram showing the caption information creation device 10 according to the present embodiment.

  The caption information creation device 10 includes a timing unit character string extraction unit 100, a timing information determination unit 107, a caption group creation unit 105, and a timing associated information storage unit 108.

  The timing unit character string extraction unit 100 extracts a timing unit character string from the caption character string group of the original caption information.

  The timing information determination unit 107 determines timing information for each timing unit character string extracted by the timing unit character string extraction unit 100.

  When the timing information determination unit 108 determines the timing information, the timing associated information storage unit 108 stores the determination process as timing associated information in association with each timing information.

  The caption group creation unit 105 creates a caption group using the timing associated information stored in the timing associated information storage unit 108.

(8) Creation of caption group using timing-accompanying information The operation of the caption group creation unit 105 is the same as that of the caption group creation unit 105 described above.

  However, the subtitle group creation unit 105 refers to the timing incidental information stored in the timing incidental information storage unit 108 and prioritizes words having timing information based on a determination process that is less likely to include a temporal error. Create a subtitle group to be at the front of the subtitle group.

  Conversely, a subtitle group is created by avoiding placing a word having timing information based on a determination process having a high possibility of including a time error at the front end of the subtitle group.

(9) Effect According to the present embodiment, for each determined timing information, a subtitle group is created in consideration of the possibility of an error in timing information caused by a difference in the determination process. Alternatively, it is possible to calculate timing information of each timing unit character string positioned between the rear end and to create a caption group having timing information with little deviation from the actual speech timing.

  Hereinafter, a caption information creating apparatus 10 according to a fourth embodiment of the present invention will be described with reference to FIGS. In this embodiment, the caption information creating apparatus 10 is applied to English.

(1) Configuration of Subtitle Information Creation Device 10 FIG. 11 is a block diagram showing the caption information creation device 10 according to the present embodiment.

  The caption information creating apparatus 10 includes a timing unit character string extraction unit 100, an audio base timing information determination unit 101, an isochronous segment base timing information determination unit 108, a difference time base timing information determination unit 109, and a caption group creation unit 105. I have.

  The timing unit character string extraction unit 100 extracts a timing unit character string from the caption character string group of the original caption information.

  The voice base timing information determination unit 101 associates the timing unit character string with the voice, and determines timing information of each timing unit character string based on the result.

  The isochronous segment base timing information determination unit 108 determines the timing information of the undecided timing unit character string based on the number of isochronous segments and the timing information of the determined timing unit character string.

  The difference time base timing information determination unit 109, when there is an isochronous segment undecided character string between the determined timing unit character string and the undecided timing unit character string, the timing information of the determined timing unit character string, Based on the timing information of the original caption information, the timing information of the pending timing unit character string is determined.

  The caption group creation unit 105 creates a caption group.

(2) Operation of Subtitle Information Creation Device 10 FIG. 12 is a flowchart of the caption information creation device 10 according to the fourth embodiment.

  Hereinafter, the operation of the caption information creating apparatus 10 according to the present embodiment will be described with reference to the flowchart of FIG. Here, as in the first embodiment, a description will be given assuming a television having a function of displaying a CC as an example. The audio and the original subtitle information are received as broadcast radio waves, and the audio and original subtitle information extracted from this are input to the subtitle information creating apparatus 10 of the present invention.

(3) Timing Unit Character String Extraction First, in step S0 of FIG. 12, the timing unit character string extraction unit 100 extracts a timing unit character string from a caption character string group included in the original caption information.

  The individual subtitle character strings constituting the subtitle character string group are displayed on the screen at a time in synchronism with the audio, and display timing information of display start and end is respectively displayed as shown in c1 to c5 of FIG. Accompanying. Each of the display start time, display end time, and subtitle character string is indicated by “,” delimiters. The display start and end times are in seconds with the audio playback start time being 0 seconds.

  The timing unit character string to be extracted is a fixed character pattern. In the caption information creation device 10, the first is the association element on the caption character string side when the speech and the caption character string are associated, and the second is the caption group. It has the role of leading edge candidate. Therefore, it is desirable that the character pattern defined as the timing unit character string is a character string composed of as few characters as possible for association with audio, and is a unit that can be the front end of a caption group. And

  In the first embodiment, the subtitle information creation device 10 is applied to Japanese and “timing unit character string” is used as a word. However, in this embodiment, since English is used as an example, the timing unit character string is a foot. Then, the timing unit character string extraction unit 100 divides the timing unit character string for each foot.

  A “foot” is an utterance unit that is divided for each stress in an English utterance. A stress is a syllable that is spoken strongly and slowly, and a word in the sentence is a stressed word (hereinafter referred to as a “stressed word”) and a word that is spoken quickly without a stress (hereinafter referred to as a “stressed word”). ")". A group of stress words and a series of stress words separated by the stress words is called a foot. Since the foot is an utterance section that is uttered at a breath, it is suitable as a unit for associating with the voice. Also, since it is a group that has meaning in utterance, it is also suitable as the front end of a caption group.

  A foot is a segmental unit of sound having isochronism in English utterances, like a mora in Japanese. In the following example, capital letters indicate the position of stress, and “|” indicates the foot boundary.

1) the ｜ HOUSE is ｜ BEAUtiful
2) the ｜ HOUSE is very ｜ BEAUtiful
3) the ｜ HOUSE is not very ｜ BEAUtiful

1) to 3) “house is”, “house is very” and “house is not very” do not change the time interval between stresses regardless of the number of syllables contained in the foot (Takubo, Maekawa, Kuboen) , Honda, Shirai, Nakagawa: "Language Science 2: Speech": Iwanami Shoten: 1988). In the following description, the foot unit is used as the timing unit character string, and the foot-unit character string obtained from the caption character string is hereinafter referred to as a caption element.

(4) Audio Base Timing Information Determination Next, the audio base timing information determination unit 101 extracts a character string for comparison (hereinafter referred to as “audio element”) from the received audio and associates it with a subtitle element string. For the extraction of the voice element, voice recognition, which is known as a foot like the caption element, is used. A known dynamic programming method or the like is used for associating an audio element with a caption element sequence.

  In step S1 of FIG. 12, the timing information of the audio element is given as the timing information of the caption element (hereinafter referred to as “matching element”) in which the corresponding audio element is found. For example, as shown in FIG. 14, “house is not” on the audio element string is associated with “house is not” on the caption element string, and is 60.88 (timing information on display start time). ) And 61.08 (display end time) are given as timing information on the caption element side.

  In step S2, it is determined whether timing information of all subtitle elements has been determined after the above assignment.

  If the timing information is determined in step S2, a caption group is created and the process ends.

  The speech element sequence in FIG. 14 is a list of speech recognition results and includes recognition errors. In the example, “does” and “boots” are speech elements of recognition errors.

(5) Isochronous segment base timing information determination If there is a caption element for which timing information has not been determined (hereinafter referred to as “mismatch element”) in the determination in step S2, isochronous segment base timing information determination is performed in step S3. The unit 109 determines the timing information of the mismatched elements.

  The determination of isochronous segment base timing information will be described with reference to FIG.

  The isochronous segment base timing information determination unit 109 calculates the timing information of subtitle elements by analogy calculation based on the timing information of matching elements and the number of feet.

  For example, when determining the timing information of the non-matching element “say.” Behind the matching element “beautiful as you”, it is calculated as follows based on the time per foot (hereinafter referred to as “foot time length”) ft. To do.

ft = utterance end time of “beautiful as you” −utterance start time of “beautiful as you” = 61.48−61.28 = 0.2
"Say" utterance start time = "beautiful as you" utterance end time = 61.48
“Say” utterance end time = “say” utterance start time + ft = 61.68

One foot isochronism shall be maintained within a sentence segment delimited by periods. When different time lengths (ft) are obtained from a plurality of matching elements in one sentence, the average value of them is used, or the foot time length of the matching element closest to the non-matching element for determining timing information is used. Etc.

(6) Determination of differential time base timing information There is a character string (hereinafter referred to as “isochronous segment undecided element”) in which the number of isochronous segments cannot be uniquely determined. The first is a numeric character string that does not know whether it is a bar or digit reading, and the second is a character string that is not supported by the isochronous segment determination process. Note that a character string that is not supported corresponds to a character string that is not registered in a dictionary (when a dictionary is used for isochronous segment determination).

  In the example of FIG. 16, “10963” is an isochronous segment pending element.

“10963” is 4 feet when speaking “ten / nine / sixty / three” (“/” is the foot boundary), and 5 feet when speaking “one / o / nine / six / three”. Become. As described above, the actual utterance with respect to the notation of CC depends on the context, and further knowledge and calculation amount are required for automatic determination. If the calculation is determined by any one of the reading methods and the reading method is different from the actual reading method of the utterance, a large error may occur.

  Since there is an increased possibility of an error every time isochronous segment pending elements are mixed, there are cases where only the isochronous segment base timing information determination unit 109 cannot calculate timing information with a small error.

  Therefore, when determining the isochronous segment base timing information in step S3, if there is an isochronous segment pending element between the matching element and the mismatching element on the caption element sequence, the differential time base timing is determined in step S4. The determination unit 110 determines the timing information of the mismatched elements.

  The difference time base timing information determination unit 110 uses the timing information of the original caption information. The timing information of the original subtitle information is given for each subtitle character string in one screen display unit in the original subtitle information. These display start times are regarded as display start times of words corresponding to the front end of each subtitle character string. For example, the display start time of the subtitle character string of c2 in FIG. 13 is 27.48, which is regarded as the display start time of the front end word “decided” of the subtitle character string of c2 (e2). Similarly, the words e1, e3, and e4 are words having a display start time (hereinafter referred to as original display start time) based on the original timing information.

  The flow of the difference time base timing information determination will be described with reference to FIG.

  First, as indicated by the circled numeral 1 in FIG. 16, the timing information of the element “after” having the display start time based on the original timing information with the matching element “pound” as the base point is determined by the isochronous segment base timing information determination. calculate. Although “after” is not originally 1 foot, there is an isochronous segment undetermined element in the front, so it is one element for convenience.

  Next, as indicated by the circled number 2 in FIG. 16, the difference between the original display start time 71.03 of “after” and the display start time 63.53 of the timing information calculated by the isochronous segment base timing information determining means. Calculate time 7.5.

  Next, as shown by a circled number 3 in FIG. 16, this difference time is subtracted from the original display start time 69.56 of “Just dial” to obtain a display start time of 62.06.

  With the above processing, the timing information of elements adjacent in both the front and rear ends is found, and the timing information of the isochronous segment undetermined element “10963” is determined.

(7) Subtitle group creation As shown in FIG. 12, the audio base timing information is determined in step S1, the isochronous segment base timing information is determined in step S3, and the differential time base timing information is determined in step S4. Once the timing information is determined, the subtitle group creation unit 105 next creates a subtitle group in step S6.

  The creation follows firstly the timing information determined by (S1) to (S4), secondly the user's desire for the subtitle display form, and thirdly the format considering readability.

  If the display end time of a certain element is not the same as the display start time of the next element in the determined timing information, these two elements are not set as the same caption group.

  The user's request for the subtitle display form may be to specify the number of lines or the number of characters.

In addition, the format considering readability is
・ Determine the number of characters to be displayed at one time and the display time in consideration of statistically obtained human reading speed.

    -Make sure that the end of the display character string (..), comma (","), question mark ("?"), Exclamation mark ("!"), Etc. does not come.

    ・ No separation immediately after “Mr.”, “Ms.”, “Mt.”, etc.

  And so on.

  Finally, the display start time of each subtitle group front end foot is the subtitle group display start time, and the subtitle group rear end foot display end time is the subtitle group display end time. Regarding how to determine the display start and end times of the subtitle group, it may be possible to intentionally give a time difference based on user requests and readability. For example, as shown below, the subtitle utterance time is set longer than the actual utterance time to cope with a case where the assumed subtitle reading speed of the user is slower than the actual utterance.

    ・ If there is a certain time difference Td between the display end time of the subtitle group g and the display start time of the subtitle group displayed next to g, an addition of Td or less is added to the display end time of the subtitle group g. Thus, the subtitles are displayed for a certain time after the utterance corresponding to the subtitle group N ends.

    ・ If there is a certain time difference Td between the display start time of the subtitle group g and the display end time of the subtitle group displayed before g, subtract Td or less from the display start time of the subtitle group g. Thus, the subtitle group g is displayed from a predetermined time before the utterance corresponding to the subtitle group g starts.

(8) Effects According to the present embodiment, in English as well as Japanese using mora, timing information can be calculated, a subtitle group having these character strings as the front end can be created, and the CC display state can be improved. Can respond to requests.

  Hereinafter, a caption information creating apparatus 10 according to a fifth embodiment of the present invention will be described with reference to FIGS.

  The present embodiment estimates the number of isochronous segments within the isochronous segment undetermined range, and includes an isochronous segment estimation unit 111. The undecided range is a range in which the front and rear ends are isochronous segment undecided elements, and there is no matching element or caption element having the original display start time in the range. However, if multiple ranges are adjacent or overlap, only the maximum range that combines them is allowed.

(1) Configuration of Subtitle Information Creation Device 10 FIG. 17 is a block diagram showing the caption information creation device 10 according to the present embodiment. Only a different part from FIG. 11 of Example 4 is demonstrated.

  The isochronous segment estimation unit 111 estimates the number of feet in the range for the undecided range.

  The isochronous segment base timing information determination unit 109 determines the timing information of each subtitle element within the range based on the estimated number of feet.

(2) Isochronous Segment Estimation With reference to FIG. 18, isochronous segment estimation will be described.

  The three elements “CHI”, “or”, and “ISWC” on the caption element string constitute one undecided range. In this case, it is expected that “next” adjacent to the rear of the range (which is a single element for convenience because it has a pending range ahead but not a “foot”) will be part of the rear foot of the pending range. The difference between the display end time of “it to” and the display start time of “year” is defined as the total elapsed time of the target range.

  In the example 72.79-72.19 = 0.6 is obtained. Next, all combinations of foot candidates for each element within the range are created, and the time per foot is calculated for each combination based on the total time of the range.

  “CHI” is an abbreviation for “Computer Human Interaction” and is the world's largest academic society related to user interfaces. This is generally spoken by “Kai” (1 foot) or “Chi, H, Eye” (3 feet) one character at a time.

  “ISWC” is an abbreviation for “International Symposium on Wearable Computers” and is spoken by “Easwick” (1 foot) or “I, S, W, Sea” (4 foot). Therefore, the number of combinations is 2 × 2 = 4.

  From the calculated time, one is selected based on the time per foot before and after the pending range. In the example, 0.3 seconds per foot is obtained from the foot “send it to” in front of the range. Therefore, “Kai” (1 foot) + “Easwick” (1 foot), which is a combination of the number of feet for which 0.3 is calculated as the time per foot, is selected and used as the estimation result. As the selection method, for example, the time per foot before and after the pending range may be averaged, and the time closest to the value may be selected.

  Based on the obtained estimation result, the foot configuration of the caption element sequence in FIG. 18 and the respective timing information can be determined as shown in FIG.

Example of change

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, constituent elements over different embodiments may be appropriately combined.

  For example, in each of the above embodiments, the language is applied to Japanese and English, but the present invention is also applicable to German, French, Italian, Spanish, Russian, Chinese, Korean, etc. it can.

1 is a block diagram of a caption information creating device 10 according to a first embodiment of the present invention. 3 is a flowchart showing the operation of the first embodiment. It is an example of original subtitle information and a timing unit character string. It is explanatory drawing of audio | voice base timing information determination. It is explanatory drawing of mora base timing information determination. It is explanatory drawing of difference time base timing information determination. It is a block diagram of the closed caption information creation apparatus 10 of Example 2. It is explanatory drawing of mora number estimation. It is a block diagram of the closed caption information creation apparatus 10 of Example 3. It is a figure of the state which displayed subtitles on the television. It is a block diagram of the closed caption information creation apparatus 10 of Example 4. 10 is a flowchart illustrating the operation of the fourth embodiment. It is an example of original subtitle information and a timing unit character string. It is explanatory drawing of audio | voice base timing information determination. It is explanatory drawing of isochronous segment base timing information determination. It is explanatory drawing of difference time base timing information determination. It is a block diagram of the closed caption information creation apparatus 10 of Example 5. It is explanatory drawing of isochronous segment estimation. It is a table | surface of the foot structure of FIG. 17, and timing information based on the result of isochronous segment estimation.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Subtitle information creation apparatus 100 Timing unit character string extraction part 101 Voice base timing information determination part 102 Mora number determination part 103 Mora base timing information determination part 104 Difference time base timing information determination part 105 Subtitle group creation part 106 Mora number estimation part 107 Timing information determination unit 108 Timing information storage unit 109 Isochronous segment base timing information determination unit 110 Differential time base timing information determination unit 111 Isochronous segment estimation unit

Claims (8)

In a caption information creation device that creates caption information for audio,
Receiving original subtitle information including original timing information, which is time information indicating a reference correspondence with the voice provided for each arbitrary number of characters in the subtitle character string group and the subtitle character string created in advance, An extraction unit that extracts a timing unit character string that gives timing information that is time information for displaying the subtitle character string in synchronization with the voice from the subtitle character string group;
First timing information determination is performed by associating the voice with the timing unit character string by voice recognition, and determining the timing information based on time information of the voice for the timing unit character string with which the correspondence has been made. And
Based on the timing information of the determined timing unit character string for which the timing information is determined, using the number of isochronous segments that are isochronous segments included in each timing unit character string, A second timing information determination unit that determines the timing information of an undecided timing unit character string for which timing information could not be determined;
When an isochronous segment undecided character string that is the timing unit character string in which the number of isochronous segments cannot be uniquely determined exists between the pair of timing unit character strings having the original timing information, (1) The difference time between the timing information of the determined timing unit character string and the original timing information is obtained, and (2) the timing information is obtained by subtracting the difference time from the original timing information of the pair of timing unit character strings. (3) a third timing information determination unit that determines the timing information of the isochronous segment undecided character string between them using the timing information of the pair of timing unit character strings;
A subtitle group that is a subtitle character string obtained by dividing the subtitle character string group according to an arbitrary format, the determined timing unit character string for displaying each subtitle group in synchronization with the audio, and the undecided timing unit A creation unit for creating the subtitle information including a character string or the timing information of each of the isochronous segment undecided character strings;
A subtitle information creating apparatus characterized by comprising: Finding an undecided range in which the front and rear ends of the subtitle character string group are the isochronous segment undecided character string and the character string group not including the determined timing unit character string,
Based on a combination of the timing information of the determined timing unit character strings adjacent before and after the undecided range and isochronous segment candidates of the timing unit character strings in the undecided range, 1 isochronous for each combination Calculate the display time per segment,
Choose the combination of isochronous segment candidates that gives the closest time compared to the display time per isochronous segment before and after the pending range;
An estimation unit that estimates the number of selected isochronous segment candidates as the number of isochronous segments of the timing unit character string within the pending range;
The closed caption information creating apparatus according to claim 1. Timing associated information indicating that the timing information is determined by the first timing information determination unit, the second timing information determination unit, or the third timing information determination unit is associated with each timing information. A timing-accompanying information storage unit for storing
The creation unit sets the timing unit character string having the timing information with a small time error based on the timing-accompanying information at a front end of the caption group.
The closed caption information creating apparatus according to claim 1. The second timing information determining unit includes an isochronous segment number determining unit that determines the number of isochronous segments included in the timing unit character string.
The closed caption information creating apparatus according to claim 1. The extraction unit divides the timing unit character string with the isochronous segment as a minimum constituent unit.
The closed caption information creating apparatus according to claim 1. The isochronous segment is a mora or a foot.
The closed caption information creating apparatus according to claim 1. In the caption information creation method for creating caption information for audio,
Receiving original subtitle information including original timing information, which is time information indicating a reference correspondence with the voice provided for each arbitrary number of characters in the subtitle character string group and the subtitle character string created in advance, An extraction step of extracting a timing unit character string that gives timing information, which is time information for displaying the subtitle character string in synchronization with the voice, from the subtitle character string group;
First timing information determination is performed by associating the voice with the timing unit character string by voice recognition, and determining the timing information based on time information of the voice for the timing unit character string with which the correspondence has been made. Steps,
Based on the timing information of the determined timing unit character string for which the timing information is determined, using the number of isochronous segments that are isochronous segments included in each timing unit character string, A second timing information determination step for determining the timing information of an undecided timing unit character string for which timing information could not be determined;
When an isochronous segment undecided character string that is the timing unit character string in which the number of isochronous segments cannot be uniquely determined exists between the pair of timing unit character strings having the original timing information, (1) The difference time between the timing information of the determined timing unit character string and the original timing information is obtained, and (2) the timing information is obtained by subtracting the difference time from the original timing information of the pair of timing unit character strings. (3) a third timing information determining step of determining the timing information of the isochronous segment undecided character string between them using the timing information of the pair of timing unit character strings;
A subtitle group that is a subtitle character string obtained by dividing the subtitle character string group according to an arbitrary format, the determined timing unit character string for displaying each subtitle group in synchronization with the audio, and the undecided timing unit Creating the subtitle information including a character string or the timing information of each of the isochronous segment undecided character strings;
A method for creating subtitle information, comprising: On the computer,
Receiving original subtitle information including original timing information which is time information indicating a reference correspondence with a subtitle character string group created in advance and audio given for each arbitrary number of characters in the subtitle character string, An extraction function for extracting from the subtitle character string group a timing unit character string that gives timing information that is time information for displaying the subtitle character string in synchronization with voice;
First timing information determination for associating the speech with the timing unit character string by speech recognition and determining the timing information based on the time information of the speech for the timing unit character string that has been associated Function and
Based on the timing information of the determined timing unit character string for which the timing information has been determined, using the number of isochronous segments that are isochronous sound segments included in each timing unit character string, A second timing information determination function for determining the timing information of an undecided timing unit character string for which timing information could not be determined;
When an isochronous segment undecided character string that is the timing unit character string in which the number of isochronous segments cannot be uniquely determined exists between the pair of timing unit character strings having the original timing information, (1) The difference time between the timing information of the determined timing unit character string and the original timing information is obtained, and (2) the timing information is obtained by subtracting the difference time from the original timing information of the pair of timing unit character strings. (3) a third timing information determining function for determining the timing information of the isochronous segment undecided character string between them using the timing information of the pair of timing unit character strings;
A subtitle group that is a subtitle character string obtained by dividing the subtitle character string group according to an arbitrary format, the determined timing unit character string for displaying each subtitle group in synchronization with the audio, and the undecided timing unit A creation function for creating the subtitle information including a character string or the timing information of each of the isochronous segment undecided character strings;
Subtitle information creation program for realizing

Images