JP2010233019A - Caption shift correction device, reproduction device, and broadcast device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To sequentially create a broadcast content which corrects a time shift between a video or a voice and a caption with a high degree of accuracy while receiving the broadcast content. <P>SOLUTION: A voice recognition unit 21 recognizes the voice of a broadcasting program and generates a recognition result phoneme sequence corresponding to the voice. A caption conversion phoneme sequence generation unit 22 generates the phoneme sequence corresponding to each caption in the video of the broadcasting program and generates a caption conversion phoneme sequence by connecting these phoneme sequences. A phoneme sequence collation unit 23 associates the caption with the voice by using information of the voice and the caption in the received broadcast content at a constant time interval, predicts a start time and an end time after correction of the time shift with respect to the caption based on the result, and estimates a range of the time shift of the voice and the caption. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a caption displacement correction device, a playback device, and a broadcast device, and in particular, can accurately correct a temporal displacement width between video and audio included in broadcast content and captions, and search for a specific video portion from captions. The present invention relates to a caption shift correction apparatus, a playback apparatus, and a broadcast apparatus that can transmit broadcast content including video, audio, and captions that can be played back and whose temporal shift width is corrected with high accuracy.

  In recent years, it has been recommended to add subtitles to TV broadcast program videos, and it has become possible to easily watch TV broadcast programs with subtitles as terrestrial digital broadcasting has started. . As a result, TV broadcast programs with subtitles tend to increase.

  When a TV broadcast program has subtitles, a procedure is generally taken in which the utterances of the performers are viewed, the contents are converted into text, output as subtitles, and added to the video. However, since it takes time to transcribe the content of the performer's utterance and create a subtitle, the time when the subtitle is output is delayed from the utterance of the performer, and there is a time lag between the video and audio and the subtitle. Arise.

  FIG. 13 shows this state. For example, performers is "Hello. This morning the news. ..." If you say, subtitle is output with a delay time required for the character of the speech content. For this reason, a time lag occurs between video and audio and subtitles. Subtitles may be output during the utterance.

  In Patent Document 1, in order to eliminate the difference in display timing between video and subtitles in a broadcast program, the time lag between the audio and the broadcast manuscript is estimated in the broadcast station, and subtitles are output based on the time lag. An apparatus for determining timing is described.

  In recent years, efforts have been made to use subtitles as meta information for video search. Patent Document 2 discloses a method for estimating a time lag between audio and subtitles on a broadcast receiving side, correcting a timecode of subtitles, and generating meta information used for search, and video search using the meta information. An apparatus for performing is described.

  In addition, the present inventor has proposed a subtitle deviation estimation device that estimates a temporal deviation width between audio and subtitles using a subtitle / audio collation method that does not depend on the utterance content in Patent Documents 3 and 4 (prior application). In the caption shift estimation device of Patent Document 3, a temporal shift width is predicted based on the length of the caption, the collation range between the caption and the voice is limited, and the collation result is weighted, thereby performing calculation processing related to collation The amount is reduced and the estimation accuracy of the time lag is improved. In the caption shift estimation device of Patent Document 4, the phoneme sequence obtained from the speech recognition result and the phoneme sequence obtained by converting the caption are collated over the entire broadcast content, and the optimum correspondence between the caption and the speech is performed. This improves the estimation accuracy of the time lag width between the audio and the caption.

JP-A-10-136260 JP 2005-229413 A Japanese Patent Application No. 2007-236550 (prior application) Japanese Patent Application No. 2008-093029 (prior application)

  As described above, there are a method for correcting the time lag between video and audio and subtitles, and a method for improving the accuracy of search meta information by correcting the time lag between video and audio and subtitles. Although it has been proposed, for the correction, it is required to estimate the temporal shift width with a small amount of calculation processing and with high accuracy.

  In the caption shift estimation device of Patent Document 1, an audio model representing a phonetic symbol string corresponding to the beginning portion of a broadcast manuscript used at the time of broadcast recording is collated with speech, and the speech section having the highest collation score with the beginning portion of the broadcast manuscript And the subtitle output time is determined based on the detected time information of the voice section. However, when this method is applied to associating video and audio with subtitles, it is necessary to perform a collation process between the acoustic model and audio over a wide range, and there is a problem that a large amount of calculation processing is required. In addition, when there are a plurality of voices similar to the head part of the caption, it is difficult to specify the voice section corresponding to the caption.

  Patent Document 2 does not describe a specific method for associating video and audio with captions. This describes that, when estimating the time lag between audio and subtitles, the amount of calculation processing is reduced by determining and collating the time lag range. When it is expected to cover a wide range, there is a problem that the amount of calculation processing required for collation increases and collation accuracy also decreases.

  Even in the caption shift estimation device of Patent Document 3, the amount of calculation processing required to estimate the time shift width between the voice and the caption is reduced by determining and collating the time shift range. There is. In addition, since the time lag between audio and each subtitle is estimated independently, multiple subtitles may be associated with the same audio part in duplicate, or the order of subtitle association may be reversed. is there. Therefore, when the time lag between the audio and the subtitles is corrected according to the estimated time lag width, there is a possibility that the subtitles before and after are overlapped in time or the order of the subtitles before and after is reversed. .

  In the caption shift estimation device of Patent Document 4, the entire phonetic content is collated with the phoneme sequence obtained from the speech recognition result and the phoneme sequence obtained by converting the caption, and the subtitle and the speech are optimally matched. It is possible to prevent deterioration of correction accuracy as in Patent Document 3. However, since it is necessary to correct the time lag after acquiring all subtitles and audio in the target broadcast content, the time lag between video and audio and subtitles is received while receiving the broadcast content. There is a problem that real-time correction and the use of the correction result cannot be performed, such as correcting the image and reproducing the result.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems, and capable of sequentially creating broadcast content in which broadcast content is received and broadcast content in which temporal deviation between video and audio and caption is corrected with high accuracy is created. Another object is to provide a correction device, a playback device, and a broadcasting device.

  In order to solve the above-described problem, a caption shift correction apparatus according to the present invention recognizes a sound in a received broadcast content while receiving the broadcast content, and generates a recognition result phoneme string corresponding to the sound. And a phoneme sequence corresponding to each subtitle in the video of the broadcast content, and a subtitle conversion phoneme sequence generation unit that generates a subtitle conversion phoneme sequence by concatenating those phoneme sequences, and generated by the speech recognition unit A phoneme string collating unit that associates subtitles with audio based on an edit distance between the recognition result phoneme string and the caption converted phoneme string generated by the caption converted phoneme string generating unit, and determines the start and end times of the subtitles And a shift correction unit that corrects a time shift between the audio and the subtitle based on the start and end times of the subtitles determined by the phoneme sequence collation unit, and the phoneme sequence collation unit And character Corresponding with the sound before reception, predict the start and end time of the caption after time lag correction from this result, and then at regular intervals until the predicted start and end time of the caption, The process of associating subtitles with audio using the audio and subtitle information in the newly received broadcast content, and repeating the process of predicting the start and end times of the subtitles after correcting the time lag from the result is as follows. There is one feature.

  Further, the subtitle shift correcting apparatus according to the present invention is such that the phoneme string collating unit determines the result of associating the subtitle with the audio before receiving the subtitle, the predicted start time of the subtitle, and the subtitle acquired when the broadcast content is received. There is a second feature in that the end time of the caption after the time lag correction is predicted based on the display time.

  In the caption shift correcting device according to the present invention, the phoneme string matching unit applies the subtitle estimated from the result of associating the caption with the audio before the caption reception, the predicted start time of the caption, and the caption character string. A third feature is that the end time of the caption after correcting the time lag is predicted based on the predicted value of the corresponding voice length.

  Further, the caption shift correction device according to the present invention stores the recognition result phoneme sequence used when the phoneme sequence verification unit predicts the start and end times of the caption after time shift correction, and the phoneme sequence verification The unit stores the recognition result phoneme sequence generated by the speech recognition unit from the time of the previous processing to the current time when repeating the process of predicting the start and end times of the subtitles after the time lag correction. The recognition result phoneme string is combined with the recognized recognition result phoneme string to generate a recognition result phoneme string, and the end time of the caption after time lag correction based on the edit distance between the recognition result phoneme string generated thereby and the caption conversion phoneme string There is a fourth feature in predicting.

  Further, the subtitle shift correcting apparatus according to the present invention includes a recognition result phoneme sequence used when the phoneme sequence matching unit predicts a start time and an end time of a subtitle after time shift correction and a recognition intermediate recognized by the speech recognition unit. The result and the collation intermediate result collated by the phoneme string collation unit are stored, and the speech recognition unit performs the process of predicting the start and end times of the subtitles after the time lag correction at the time of the previous process. Recognizing the speech from the time of the previous processing to the current time taking over the stored recognition intermediate result and generating a recognition result phoneme sequence, the phoneme sequence collation unit recognizes the recognition result phoneme sequence stored The recognition result phoneme string is generated by combining with the result phoneme string, and the end time of the caption after the time lag correction is predicted using the generated recognition result phoneme string and the collation intermediate result saved at the previous processing. The fifth feature A.

  Further, the subtitle shift correcting apparatus according to the present invention includes a recognition result phoneme sequence used when the phoneme sequence matching unit predicts a start time and an end time of a subtitle after time shift correction and a recognition intermediate recognized by the speech recognition unit. The result and the collation intermediate result collated by the phoneme string collation unit are stored, and the speech recognition unit performs the process of predicting the start and end times of the subtitles after the time lag correction at the time of the previous process. Recognizing the speech from the time of the previous processing to the current time by taking over the stored recognition intermediate result and generating a recognition result phoneme string, the phoneme string collation unit first recognizes this recognition result phoneme string Compare the recognition result phoneme sequence that recognized the speech from the start time to the current time, and go back to the different phonemes, and edit the distance between the recognition result phoneme sequence and the subtitle conversion phoneme sequence, the stored collation intermediate result Calculate using the Distance is the sixth aspect of the point to estimate the temporal deviation of the audio and subtitles based on.

  In addition, the caption shift correction apparatus according to the present invention estimates a time gap between audio and caption, and then recognizes a caption corresponding to a speech section that is earlier than a predetermined time from the estimated time and a corresponding caption. The seventh feature is that the result phoneme string, the subtitle converted phoneme string, and the collation intermediate result are discarded.

  In addition, the caption shift correcting device according to the present invention estimates a temporal shift width between speech and caption, and then recognizes a recognition result phoneme and a caption converted phoneme sequence corresponding to a speech section that is a predetermined time or more before the estimated time. There is an eighth feature in that the collation intermediate result is discarded.

  Further, the caption displacement estimation device according to the present invention uses the cost corresponding to the number of steps required to convert the edit distance from one phoneme sequence to the other as a result of insertion, deletion, and replacement of phonemes. There is a ninth feature in the point defined.

  In addition, the caption displacement correction device according to the present invention is configured such that the editing distance is determined based on the speech recognition performance for each phoneme, and the cost required for replacing a phoneme with a phoneme, when inserting a phoneme. The tenth feature is that the cost required for deleting a certain phoneme is defined as an index.

  Further, in the caption shift correction device according to the present invention, the caption conversion phoneme sequence generation unit generates a caption conversion phoneme sequence in which pseudo phonemes representing sentence breaks are added to caption breaks, and the phoneme sequence matching unit includes: An eleventh feature is that the pseudo-phonemes are given a cost smaller than that of other phonemes and the edit distance is calculated.

  Further, in the caption shift correcting device according to the present invention, the caption converted phoneme string generation unit generates a caption converted phoneme string in which a pseudo phoneme representing a sentence break is added to a sentence break obtained by analyzing the caption, The phoneme string collation unit has a twelfth feature in that the pseudo-phoneme is given a cost smaller than that of other phonemes and calculates the edit distance.

  Further, in the caption shift correcting device according to the present invention, the caption converted phoneme string generating unit adds a pseudo phoneme representing a sentence break to a sentence break obtained by analyzing the caption break and the caption. There is a thirteenth feature in that the phoneme string collating unit calculates the edit distance by giving the pseudo-phoneme a cost smaller than that of other phonemes.

  Further, in the caption shift correcting device according to the present invention, the caption converted phoneme string generation unit generates a caption converted phoneme string in which a pseudo phoneme representing a sentence break is added to a sentence break obtained by analyzing the caption, The speech recognition unit generates a recognition result phoneme sequence in which a pseudophoneme representing silence is added to a place where silence continues for a certain period of time, and the phoneme sequence matching unit is a pseudophoneme representing a pseudophoneme representing silence and a sentence break The fourteenth feature is that the edit distance is calculated by setting the cost between the pseudophoneme representing silence and the other phoneme to a large value by comparing the cost between the pseudophoneme representing silence and the other phoneme.

  Further, in the caption shift correcting device according to the present invention, the phoneme string collating unit includes a first r phoneme of the recognition result phoneme string and a first n phoneme of the caption converted phoneme string (n = 1 to C: C is a caption converted phoneme). Edit distance between the total number of phonemes in the sequence), and select the top N smaller values from the edit distance, and edit between the first r + 1 phoneme of the recognition result phoneme sequence and the subtitle conversion phoneme sequence When calculating the distance, the edit distance is calculated only for the top n phonemes (n = 1 to C) of the caption conversion phoneme sequence that can be calculated using the top N selected edit distances. The fifteenth feature is that the process of not calculating the others is repeatedly executed to calculate the edit distance between the recognition result phoneme string and the caption converted phoneme string.

  Further, in the caption shift correcting device according to the present invention, the phoneme string collating unit includes a first r phoneme of the recognition result phoneme string and a first n phoneme of the caption converted phoneme string (n = 1 to C: C is a caption converted phoneme). Edit distance between the total number of phonemes in the sequence), and from the edit distance, select the smallest one and the one whose difference from the smallest one falls within a predetermined threshold, and the recognition result phoneme sequence When calculating the edit distance between the first r + 1 phoneme of the subtitle and the subtitle converted phoneme string, using the edit distance previously selected from the first n phonemes (n = 1 to C) of the subtitle converted phoneme string There is a sixteenth feature in that the edit distance is calculated only for those that can be calculated, the process of not calculating the other is repeatedly executed, and the edit distance between the recognition result phoneme string and the caption converted phoneme string is calculated.

  Further, in the caption shift correcting device according to the present invention, the phoneme string collating unit includes a first r phoneme of the recognition result phoneme string and a first n phoneme of the caption converted phoneme string (n = 1 to C: C is a caption converted phoneme). Edit distance between the total number of phonemes in the sequence), and select the first m phoneme of the subtitle conversion phoneme sequence having the smallest value from the edit distance, and the first r + 1 phoneme of the recognition result phoneme sequence and the When calculating the edit distance between caption conversion phoneme strings, the first r + 1 phoneme of the recognition result phoneme string and the first mN phoneme (N is a constant value) from the first mN phoneme of the caption conversion phoneme string The editing distance is calculated only for those that can be calculated using the editing distance with the first r phoneme that has already been calculated, and the other is not calculated. A seventeenth feature is that the edit distance between subtitle-converted phoneme strings is calculated.

  In addition, a playback device according to the present invention includes a caption shift correction device having any one of the first to seventeenth features and a caption whose time shift is corrected by the caption shift correction device in received broadcast content. There is a first feature in that it has a reproducing means for reproducing together with the audio and video.

  In addition, the playback device according to the present invention stores the caption shift correction device having any one of the first to seventeenth features and the caption in which the temporal shift between audio and video is corrected by the caption shift correction device. Subtitle storage means, and a search means for searching for a portion of the video that matches the input keyword based on character information in the subtitle stored in the subtitle storage means. A second feature is that the video is reproduced.

  In addition, the broadcast device according to the present invention is characterized in that it includes the subtitle shift correction device and a transmission unit that transmits audio, video, and subtitles whose time shift is corrected by the subtitle shift correction device as a broadcast program. is there.

  In the present invention, while receiving broadcast content, the recognition result phoneme sequence obtained as a result of recognizing the sound in the broadcast content and the subtitle converted phoneme sequence obtained by converting the subtitle are sequentially collated, and the subtitle and the audio are associated with each other. The subtitle start and end times are determined, and the time lag between the video and audio and the subtitle is corrected based on the result. At this time, when subtitles are received, the subtitles are associated with the audio before the subtitles are received, and the start and end times of the subtitles after the time lag correction are predicted from the results, and then the start of the predicted subtitles, At regular intervals until the end time is reached, subtitles and audio are associated with each other using the audio and subtitle information in the newly received broadcast content, and the start and end of the subtitles after correcting the time lag from the result The process of predicting the time is repeated. In this way, by sequentially determining the start and end times of the subtitles in the broadcast content, the time lag between the video and audio and the subtitles can be accurately corrected while receiving the broadcast content. Broadcast contents in which the time lag is corrected with high accuracy can be sequentially created.

  In addition, according to the eleventh to fourteenth features, by associating audio and subtitles in consideration of subtitle breaks, subtitle sentence breaks, and utterance breaks in audio within broadcast content, The time lag between the audio and the subtitle can be corrected with high accuracy.

  Further, according to the seventh, eighth, fifteenth to seventeenth features, it is possible to reduce the amount of memory and the amount of calculation required for correcting the time lag between the video and audio and the caption.

It is a block diagram which shows the basic composition of the caption shift correction apparatus which concerns on this invention. It is a block diagram which shows one Embodiment of a caption start / end time determination part. It is a block diagram which shows the structural example of a speech recognition part. It is a figure which shows the example of a language model. It is a schematic diagram which shows the collation process in a voice collation part. It is a schematic diagram which shows the production | generation process of a caption conversion phoneme sequence in a caption conversion phoneme sequence production | generation part. It is a figure explaining 1st Embodiment (1st step) of phoneme string collation processing. It is a figure explaining 1st Embodiment (2nd step) of phoneme string collation processing. It is a figure explaining 1st Embodiment (3rd step) of phoneme string collation processing. It is a figure explaining 1st Embodiment (4th step) of phoneme string collation processing. It is explanatory drawing which shows the example of the edit distance calculation process for reducing the calculation processing amount of deviation width estimation. It is explanatory drawing which shows the other example of the edit distance calculation process for reducing the calculation processing amount of deviation width estimation. It is explanatory drawing which shows the time relationship between the utterance of the performer of a broadcast program, and the output of a caption.

  The present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a basic configuration of a caption shift correction apparatus according to the present invention. This caption deviation correction apparatus includes an information separation unit 11, a caption start / end time determination unit 12, and a deviation correction unit 13. What is corrected here is a time lag between the audio and the caption when viewed from the information content. Hereinafter, the term “deviation” and “deviation width” also mean the temporal deviation and the magnitude of temporal deviation (temporal deviation width).

  Broadcast content is input to the information separation unit 11. Broadcast content includes video, audio, and subtitles stored in each track. From the viewpoint of information content, there is no time lag between video and audio, and subtitles have a time delay relative to them. The information separation unit 11 separates audio and subtitles in the broadcast content separately.

  The subtitle start / end time determination unit 12 determines the subtitle output start time and end time from the correspondence between the audio and subtitles separated by the information separation unit 11. The caption start / end time determination unit 12 will be described in detail later.

  Based on the start time and end time of the subtitles determined by the subtitle start / end time determination unit 12, the misalignment correction unit 13 sequentially outputs broadcast content that does not deviate from video, audio, and subtitles in terms of information content. To do. Deviations between video, audio, and subtitles are eliminated by temporarily storing each piece of information for each track storing video, audio, and subtitles in a buffer and controlling the read timing. That is, subtitles may be read out and output in accordance with the start time and end time of the subtitle determined by the subtitle start / end time determination unit 12 while reading and outputting video and audio as needed.

  FIG. 2 is a block diagram showing an embodiment of the caption start / end time determination unit 12. The subtitle start / end time determination unit of the present embodiment includes a speech recognition unit 21, a recognition result phoneme sequence storage unit 22, a subtitle conversion phoneme sequence generation unit 23, a subtitle conversion phoneme sequence storage unit 24, a phoneme sequence collation unit 25, and a phoneme sequence. A verification result storage unit 26 is provided.

  In the case of TV broadcast content, the voice uttered by the performer is acquired together with the video of the performer. Therefore, it can be considered that the video and audio are acquired without time lag. The voice thus acquired is input to the voice recognition unit 21. The speech recognition unit 21 recognizes the input speech and sends a recognition result phoneme string corresponding to the speech.

  FIG. 3 is a block diagram illustrating a configuration example of the voice recognition unit 21. The speech recognition unit 21 includes a speech detection unit 31, an acoustic analysis unit 32, an acoustic model storage unit 33, a language model storage unit 34, and a speech collation unit 35. This configuration is common in speech recognition.

  The voice detection unit 31 cuts out the voice of the section including the human voice from the input voice and sends it to the acoustic analysis unit 32. For example, a voice detection method based on the magnitude of input power can be used to cut out the voice in the voice detection unit 31. In this voice detection method, the input power is sequentially calculated, and when the input power continuously exceeds a predetermined threshold for a certain period of time, it is determined as the voice start time, and conversely, the input power is predetermined. The point of time when the threshold value is continuously lowered for a certain time is determined as the end point of the voice. The voice cut out by the voice detection unit 31 is sequentially sent to the acoustic analysis unit 32 from the voice start time to the voice end time.

  The acoustic analysis unit 32 performs an acoustic analysis of the voice cut out by the voice detection unit 31, and sends an acoustic feature amount sequence representing a voice feature such as MFCC to the voice collation unit 35.

  The acoustic model storage unit 33 stores a standard pattern such as an HMM prepared for each phoneme which is a unit constituting Japanese speech. A standard pattern corresponding to any Japanese word / sentence can be created by connecting this standard pattern in accordance with the phoneme sequence constituting the Japanese word / sentence.

  The language model storage unit 34 stores a language model that defines connection relationships such as between Japanese words and phonemes. This language model includes (1) a continuous syllable recognition grammar that prescribes the connection relationship between syllables, (2) a grammar rule that prescribes a connection relationship between words, and (3) a probability that a set of N phonemes is continuous. There are statistical language models that prescribe, and (4) statistical language models that prescribe the probability that a set of N words will continue.

  FIG. 4 is a diagram illustrating an example of a language model. Figure (a) is a continuous syllable recognition grammar that defines the connection relationship between syllables. This shows the connection relationship between consonant / b // d / ... and vowel / a // i / ... It prescribes. FIG. 4B shows grammatical rules that define the connection relationship between words, which defines the connection relationship of / word1 // word2 /. The language model is described in, for example, “Kano et al .:“ IT Text Speech Recognition System ”Ohmsha”.

  FIG. 5 is a schematic diagram showing collation processing in the voice collation unit 35 (FIG. 3). In the figure, the acoustic feature quantity sequence sent from the acoustic analysis unit 32 is collated with the standard pattern by the voice collation unit 35, thereby obtaining a collation result /sh//i/.../u/ It shows that the start and end times of the speech segment corresponding to the phoneme are acquired.

  Returning to FIG. 3, the voice collation unit 35 generates a standard pattern by connecting the acoustic model according to the connection rules described in the language model, and uses the Viterbi algorithm to transmit the acoustic feature amount from the acoustic analysis unit 32. Match columns against standard patterns. As a result of this collation, the correspondence between the voice section that maximizes the collation score of both and the standard pattern is obtained. As the recognition result in the speech recognition unit 21, the start time and the end time of the speech section corresponding to each phoneme constituting the recognition result phoneme string and the standard pattern are acquired. The recognition result phoneme string obtained in this way is stored in the recognition result phoneme string storage unit 22 (FIG. 2). Voice collation is described in “Seiichi Nakagawa et al .:“ Voice Recognition by Stochastic Model ”” The Institute of Electronics, Information and Communication Engineers.

  The recognition result acquisition process of the speech recognition unit 21 is performed when instructed by the phoneme string collation unit 25. At this time, it is assumed that the collation intermediate result (partial collation result) obtained by the voice collation unit 35 during the collation process is held. After that, according to the instruction from the phoneme string collating unit 25, the collation intermediate result at the time of the previous recognition result acquisition is continued and collation is performed. Alternatively, the collation is restarted from the initial state after discarding the collation intermediate result used at the previous collation.

  On the other hand, the caption converted phoneme string generation unit 23 generates a caption converted phoneme string (caption converted phoneme string) corresponding to the input caption. The subtitle-converted phoneme sequence generated here is not a phoneme sequence for each subtitle but a phoneme sequence in which a plurality of phoneme sequences corresponding to each subtitle in the broadcast content are connected. The caption converted phoneme string is stored in the caption converted phoneme string storage unit 24.

  FIG. 6 is a schematic diagram showing a caption conversion phoneme string generation process in the caption conversion phoneme string generation unit 23 (FIG. 2). The subtitle conversion phoneme sequence generation unit 23 performs morphological analysis of subtitles described by a kana-kana mixed sentence, divides it into parts of speech and converts it into a kana character string representing a reading, and further converts kana characters into phonetic symbols The kana character string is converted into a phoneme string by referring to the conversion table in which the conversion rules are described, thereby generating a caption converted phoneme string.

  For example, when the subtitle character string described in the kanji kana mixed sentence is “7 o'clock news”, the subtitle conversion phoneme string generation unit 23 first performs “7” “time” “no” “news” by morphological analysis. ”And“ is ”. Next, these are converted to the kana character strings "Shi", "ji", "no", "news", and "is" representing the readings, and a conversion table that describes the conversion rules from kana character strings to phonetic symbols Refer to the kana string as the phoneme string / sh / / i / / ch / / i / / j / / i / / n / / o / / ny / / uu / / s / / u / / d / / e / / s / Convert to / u /.

  The phoneme string collation unit 25 collates the recognition result phoneme string stored in the recognition result phoneme string storage unit 22 with the subtitle conversion phoneme string stored in the caption conversion phoneme string storage unit 24, and determines the subtitle Determine the output start and end times.

  The phoneme string collation result storage unit 26 holds the collation intermediate result (partial editing distance) obtained by the phoneme string collation unit 25 during the collation process. By holding a partial editing distance in the phoneme string collation result storage unit 26, a new recognition result phoneme string is added after the recognition result phoneme string, or a new caption converted phoneme string is added after the caption conversion phoneme string. It is possible to efficiently perform collation when is added. That is, when a new recognition result phoneme sequence is added after the recognition result phoneme sequence, or when a new subtitle conversion phoneme sequence is added after the subtitle conversion phoneme sequence, the partial editing process already obtained is used. The edit distance related to the newly added phoneme can be obtained. The edit distance will be described later.

  Next, the collation process in the phoneme string collation unit 25 will be described. The phoneme sequence matching unit 25 compares the recognition result phoneme sequence stored in the recognition result phoneme sequence storage unit 22 with the subtitle conversion phoneme sequence stored in the subtitle conversion phoneme sequence storage unit 24, and represents the degree of difference between the two. Calculate the distance. This editing distance is defined by using as an index the editing cost corresponding to the number of steps required to convert one phoneme string into another phoneme string by inserting, deleting, or replacing phonemes.

For example, the edit distance is the minimum number of steps required to convert one phoneme sequence to another, and "/ sh / / i / / N / / k / / a / / i /" When transforming to "/ t / / o / / k / / a / / i /", as shown below, at least three steps are required, so the edit distance is 3.
1./sh/ / i / / N / / k / / a / / i /
2. / t / / i / / N / / k / / a / / i / (Replace "/ sh /" with "/ t /")
3. / t / / o / / N / / k / / a / / i / (Replace "/ i /" with "/ o /")
4. / t / / o / / k / / a / / i / (Delete "/ N /" and exit)

  The editing distance between the recognition result phoneme string and the subtitle converted phoneme string is not limited to that defined simply according to the number of procedures. For example, based on the speech recognition performance for each phoneme (ease of error between phonemes, etc.), the cost required to replace a phoneme A with a phoneme B, the cost required to insert a phoneme A, a phoneme A It is also possible to individually define the cost required for deleting and to define the edit distance based on these costs. For example, phoneme / b / and phoneme / p / are easy to mistake, so the cost is small

  The edit distance can be calculated at high speed by using the following algorithm based on dynamic programming.

Recognition result phoneme sequence: rph [1], rph [2], ..., rph [R]
Subtitle conversion phoneme sequence: cph [1], cph [2], ..., cph [C]
From the first phoneme (rph [1]) to the last phoneme (rph [R]) in the recognition result phoneme sequence:
From the first phoneme (cph [1]) to the last phoneme (cph [C]) of a closed caption conversion phoneme sequence:
The first r phoneme rph [1 · r] of the recognition result phoneme string
Editing distance from the first c phoneme cph [1 ・ ・ c] of the caption conversion phoneme string
d [r, c] = minimum (
d [r-1, c] + ins_cost (cph [c]), // insert phoneme
d [r, c-1] + del_cost (rph [r]), // Delete phoneme
d [r-1, c-1] + sub_cost (cph [c], rph [r]) // phoneme substitution
)

  Here, ins_cost (cph [c]), del_cost (rph [r]), and sub_cost (cph [c], rph [r]) are the cost required to insert the phoneme into the recognition result phoneme string, and the recognition result, respectively. This represents the cost required to delete a phoneme from the phoneme string and the cost required to replace the phoneme in the recognition result phoneme string with another phoneme.

  This algorithm calculates the editing distance d [r−1, c] between the first r-1 phoneme rph [1 ·· r-1] of the recognition result phoneme sequence and the first c phoneme cph [1 ·· c] of the caption conversion phoneme sequence ], Edit distance d [r, c-1] between the first r phoneme rph [1 ·· r] of the recognition result phoneme sequence and the first c-1 phoneme cph [1 ·· c-1] of the caption conversion phoneme sequence, Editing distance d [r-1, c between the first r-1 phoneme rph [1 ·· r-1] of the recognition result phoneme sequence and the first c-1 phoneme cph [1 ·· c-1] of the subtitle conversion phoneme sequence -1], the edit distance d [r, c] between the first r phoneme (rph [1 ·· r]) of the recognition result phoneme sequence and the first c phoneme cph [1 ·· c1] of the caption conversion phoneme sequence The edit distance is efficiently calculated by repeatedly performing the process of calculating.

  When defining the edit distance between the recognition result phoneme sequence and the caption conversion phoneme sequence as the number of conversion steps,

Insertion cost: ins_cost (cph [c]) = always 1
Deletion cost: del_cost (rph [r]) = always 1
Replacement cost: sub_cost (cph [c], rph [r]) = 0 (when cph [c] = rph [r])
= 1 (when cph [c] ≠ rph [r])
It is.

  Simultaneously with the calculation of the edit distance, it is stored whether the selected minimum edit distance is due to the phoneme insertion, deletion, or replacement editing method. After the calculation of the edit distance to the end of the combination of the recognition result phoneme string and the subtitle conversion phoneme string, the selection result of the saved editing method is used as the last phoneme rph [R] of the recognition result phoneme string and the subtitle conversion phoneme string. By reading in reverse order from the selection result when calculating the edit distance to the last phoneme cph [C], the edit method (combination of insertion, deletion, and replacement) that minimizes the edit distance can be obtained. Based on the information, it is possible to obtain information as to which phoneme in the subtitle conversion phoneme sequence is associated with each phoneme in the recognition result phoneme sequence.

  When receiving broadcast content, a plurality of subtitles are received one after another as the broadcast content progresses. In the present invention, as described below, each time a new subtitle is received, the recognition result phoneme sequence and the subtitle converted phoneme sequence are collated sequentially to predict the start and end times of the subtitle, and based on the result In addition, while receiving broadcast content, it is possible to sequentially generate broadcast content with corrected caption displacement with a predetermined delay time T (sec) from the reception time. Note that the delay time T is determined in advance with reference to the distribution of shift widths between video and subtitles in various broadcast contents. For example, the maximum value of the deviation width between video and subtitles in all broadcast contents can be set as the delay time T (sec).

Since the present invention is particularly characterized in the recognition result phoneme string and subtitle conversion phoneme string collation process (phoneme string collation process), an embodiment of the phoneme string collation process will be described below. Here, it is assumed that the voice “Konchiwa Morning ...” is received and the corresponding subtitles are obtained with a delay.
(First embodiment)

7 to 10 are diagrams for explaining the first embodiment of the phoneme string collation processing.
(A) First, when the first subtitle “Konchiwa” is obtained, the following verification process is performed. FIG. 7 shows the collation process in this case.

  (1) Subtitle conversion phoneme sequence corresponding to subtitle conversion phoneme sequence storage unit 24 to current time "/ GB / / k / / o / / N / / n / / i / / ch / / i / / w Get / / a / / GB / ". Note that GB is a pseudo-phoneme (described later) inserted in a break between subtitle sentences.

  (2) The recognition result phoneme sequence “/ t // o // N // n / corresponding to the speech from T (sec) to the current time through the recognition result phoneme sequence storage unit 22 from the speech recognition unit 21. Get / i / ". The buffer size shown is equivalent to T (sec). Here, it is assumed that when the subtitle “Konchiwa” is acquired, the corresponding audio has not yet ended, that is, the subtitle “Konchiwa” is obtained in the middle of the utterance of “Konchiwa”. Further, the voice recognition unit 21 is instructed to collate voices after the current time after discarding the collation intermediate result.

  (3) The subtitle conversion phoneme sequence acquired in (1) and the recognition result phoneme sequence acquired in (2) are collated according to the above algorithm (collation range 1). The phoneme string matching result storage unit 26 holds the matching result for the matching range 1 as a matching intermediate result.

  (4) From the collation results up to the current time, find the phoneme in the caption converted phoneme string that has the closest edit distance from the recognition result phoneme string.

  (5) The verification history is traced back from the phonemes found in (4), and the temporary start time of subtitles is predicted.

  (6) Predict the provisional end time of subtitles according to the following criteria.

(6-1) When the phoneme predicted in (4) is the end phoneme of the subtitle conversion phoneme string, the end time of the subtitle is predicted by going back to the collation history and set as the temporary end time.
In cases other than (6-2) and (6-1), the subtitle temporary end time is predicted to be a time obtained by adding the original subtitle display time length to the subtitle temporary start time.

(B) Thereafter, the following verification process is performed at regular intervals.

(A) Until the next subtitle is obtained. That is, when the following conditions (a) to (d) are not met. FIG. 8 shows the collation process in this case.
(1) The recognition result phoneme sequence “/ ch // i // h //” corresponding to the speech from the time of the previous collation processing to the current time through the recognition result phoneme sequence storage unit 22 from the speech recognition unit 21 Get a / / a / / s / / a / ". Further, the voice recognition unit 21 is instructed to collate voices after the current time after discarding the collation intermediate result.
(2) The recognition result phoneme string acquired in (1) is added after the recognition result phoneme string used for the previous verification.
(3) The subtitle converted phoneme string used for the previous collation is collated with the recognition result phoneme string of (2) according to the above algorithm (collation range 1 + collation range 2). In this case, the collation range 2 can be collated by continuing the collation intermediate result held in the phoneme string collation result storage unit 26. The phoneme string collation result storage unit 26 further holds a collation intermediate result for the collation range 2.
(4) From the collation results up to the current time, find the phoneme in the caption converted phoneme string that has the closest edit distance from the recognition result phoneme string.
(5) The verification history is traced back from the phonemes found in (4), and the provisional start time of subtitles is predicted (updated).
(6) Predict (update) the subtitle temporary end time according to the following criteria.
(6-1) When the phoneme found in (4) is the end phoneme of the subtitle conversion phoneme string, the end time of the subtitle is predicted by going back to the collation history and set as the temporary end time.
In cases other than (6-2) and (6-1), the subtitle temporary end time is predicted to be a time obtained by adding the original subtitle display time length to the subtitle temporary start time.

(B) When the next subtitle is acquired. 9 and 10 show the collation process in this case.
(1) Add the subtitle conversion phoneme sequence "/ a / / s / / a / / n / / o / / GB /" corresponding to the subtitle acquired this time after the subtitle conversion phoneme sequence used for the previous verification ( FIG. 9).
(2) The subtitle conversion phoneme sequence of (1) is collated with the recognition result phoneme sequence used for the previous collation according to the above algorithm. Here, until the recognition result phoneme string is added, the collation range 1 + collation range2 + collation range 3 is the collation range. In the collation range 3, collation is performed on the recognition result phoneme string from T (sec) before the current time. In this case, the collation range 3 can be collated by continuing the collation intermediate result held in the phoneme string collation result storage unit 26. The phoneme string collation result storage unit 26 further holds a collation intermediate result for the collation range 3.

Further, when a recognition result phoneme string is added (FIG. 10), the collation range 1 + the collation range 2 + the collation range 3 + the collation range 4 is obtained. In this case, the collation range 4 can be collated by continuing the collation intermediate result held in the phoneme string collation result storage unit 26. The phoneme string collation result storage unit 26 further holds a collation intermediate result for the collation range 4.
(3) From the collation results up to the current time, find the phoneme in the closed caption converted phoneme string that has the shortest edit distance from the recognition result phoneme string.
(4) The verification history is traced back from the phoneme found in (3), and the temporary start time of the caption is predicted (updated).
(5) Predict (update) the temporary end time of subtitles according to the following criteria.
(5-1) When the phoneme found in (4) is the end phoneme of the subtitle conversion phoneme string, the end time of the subtitle is predicted by going back to the collation history and set as the provisional end time.
In cases other than (5-2) and (5-1), the subtitle temporary end time is predicted to be a time obtained by adding the original subtitle display time length to the subtitle temporary start time.

(C) If the subtitle provisional start time is T (sec) before the current time
(1) From the collation results up to the current time, find the phoneme in the caption converted phoneme string that has the closest edit distance from the recognition result phoneme string.
(2) The verification history is traced back from the phoneme found in (1), and the temporary start time of the caption is predicted (updated).
(3) If the newly predicted subtitle temporary start time is before or before T (sec) of the current time, the original temporary start time, that is, T (sec) before the current time, is set as the final value of the subtitle start time. .

(D) If the subtitle's provisional end time is T (sec) before the current time
(1) From the collation results up to the current time, find the phoneme in the caption converted phoneme string that has the closest edit distance from the recognition result phoneme string.
(2) The verification history is traced back from the phoneme found in (1), and the temporary end time of the caption is predicted (updated).
(3) If the newly predicted provisional end time of the caption is before or before T (sec) of the current time, the original provisional end time, that is, T (sec) before the current time, is set as the final value of the caption end time. .

According to the above collation processing, the start time and end time of subtitles can be determined at the points (c) (3) and (d) (3), respectively. Based on this result, the shift correction unit 13 (FIG. 1) corrects the shift between the video, audio, and subtitles in the broadcast content and sequentially transmits them. By repeating the above collation processing until the end of the broadcast content, the broadcast content including video, audio, and subtitles corresponding to the time is transmitted although it is delayed for a predetermined time T (sec) from the input time.
(Second Embodiment)

  In the first embodiment, the subtitle temporary end time is predicted to be a time obtained by adding the original subtitle display time length to the subtitle temporary start time (6-2). However, there is a correlation between the display time length of the subtitles and the actual utterance length when the subtitle output start time is included but the subtitle output end time is not included, or the subtitles are transmitted at regular intervals. There are low broadcast contents. In such broadcast content, it is not appropriate to predict the subtitle provisional end time by adding the original subtitle display time length to the subtitle provisional start time.

Therefore, in the second embodiment, a time obtained by adding the utterance length of the subtitle estimated from the subtitle character string to the subtitle temporary start time is predicted as the subtitle temporary end time. The utterance length can be estimated based on, for example, the caption converted phoneme string and information on the average duration of each phoneme.
(Third embodiment)

  In the first and second embodiments, after obtaining the recognition results (A-2) and (B-a-1), the speech recognition unit 21 discards the collation intermediate result, and the speech after the current time is changed from the initial state. Instructed to recognize. However, for example, when the recognition result is acquired in the middle of a certain phoneme, the speech recognition unit 21 performs recognition from the middle of the phoneme next time. On the other hand, since the recognition language model does not describe a pattern that starts from the middle of the phoneme, there is a concern that the recognition accuracy may deteriorate before and after obtaining the recognition result.

In order to avoid this problem, in the third embodiment, after obtaining the recognition results (A-2) and (B-A-1), the speech recognition unit 21 continues to take over the previous collation intermediate result. Instruct to recognize speech.
(Fourth embodiment)

  In the third embodiment, since the speech is recognized by continuing the previous collation intermediate result even after the recognition result is acquired, the recognition result in the section where the previous recognition result has been acquired may change when the next recognition result is acquired. .

  Therefore, in the fourth embodiment, in such a case, the phoneme string collation process is performed again by going back to the place where the recognition result phoneme string has changed. Specifically, the process (B) is changed as follows.

(B) Thereafter, the following verification process is performed at regular intervals.

(E) Until the next subtitle is obtained. That is, when the following conditions (a) to (d) are not met.
(1) The recognition result phoneme sequence corresponding to the speech from the beginning of the broadcast content to the current time is acquired from the speech recognition unit 21 via the recognition result phoneme sequence storage unit 22. Further, the voice recognition unit 21 is instructed to continue taking over the collation intermediate result and perform subsequent voice recognition.
(2) The recognition result phoneme string used for the previous collation is compared with the recognition result phoneme string of (1) from the beginning, and the first phoneme in which both are different is detected.
(3) The subtitle conversion phoneme sequence used for the previous verification is compared with the recognition result phoneme sequence of (1) according to the above algorithm. In this case, the collation intermediate result held in the phoneme string collation result storage unit 26 is continued and collation is started from the phoneme detected in (2). The phoneme string collation result storage unit 26 holds the collation intermediate result at this time.
(4) From the collation results up to the current time, find the phoneme in the caption converted phoneme string that has the closest edit distance from the recognition result phoneme string.
(5) The verification history is traced back from the phonemes found in (4), and the provisional start time of subtitles is predicted (updated).
(6) Predict (update) the subtitle temporary end time according to the following criteria.
(6-1) When the phoneme found in (4) is the end phoneme of the subtitle conversion phoneme string, the end time of the subtitle is predicted by going back to the collation history and set as the temporary end time.
(6-2) In cases other than (6-1), the subtitle temporary end time is predicted to be a time obtained by adding the display time length of the original subtitle to the temporary start time of the subtitle.
(Fifth embodiment)

   In the first to fourth embodiments, for the phoneme string collation, the recognition result phoneme string from the start of the broadcast content to the current time, the subtitle conversion phoneme string, and the collation intermediate result of both are held. However, the influence of the past collation results on the collation at the current time is small. Therefore, in the fifth embodiment, the amount of calculation and memory usage are reduced by discarding such information and not using them for phoneme string collation processing.

Specifically, it corresponds to a subtitle whose subtitle end time determined (confirmed) in (B-d-3) is T × α (sec) (α is a predetermined constant) or more before the current time, and that subtitle. The recognition result phoneme string, subtitle conversion phoneme string, and collation intermediate result are discarded and excluded from the calculation target of phoneme string collation.
(Sixth embodiment)

  In the fifth embodiment, unnecessary phoneme strings are discarded in units of subtitles. In the sixth embodiment, unnecessary phoneme strings are discarded in units of phonemes.

Specifically, after the subtitle end time is confirmed in (B-d-3), the end time of the recognition result phoneme is T × α (sec) (α is a predetermined constant) or more before the current time. A certain phoneme, subtitle conversion phoneme corresponding to the phoneme, and collation intermediate result are discarded and excluded from the calculation target of phoneme string collation.
(Seventh embodiment)

  In the seventh embodiment, the caption converted phoneme string generation unit 23 is provided with a function of inserting pseudo phonemes (GB) representing sentence breaks at the caption breaks when the caption converted phoneme string is generated. Then, when the editing distance is calculated by the phoneme string collating unit 25, the editing cost (insertion cost, deletion cost, replacement cost) for GB is set to a value smaller than the editing cost for other phonemes as shown below.

When cph [c] = GB Insertion cost: ins_cost (cph [c]) = 0
Deletion cost: del_cost (rph [r]) = α1 (0 <α1 <1)
Replacement cost: sub_cost (cph [c], rph [r]) = α2 ((0 <α2 <1)

In general, when speech recognition is performed, there are cases in which noise existing in silent sections such as sentence breaks is erroneously recognized and a phoneme string is output. By providing this function to the caption conversion phoneme sequence generation unit 23, it is easier to associate a phoneme sequence due to misrecognition of noise with a pseudo phoneme inserted at a caption break than a caption conversion phoneme sequence. The accuracy of association with can be improved.
(Eighth embodiment)

  In the eighth embodiment, the caption converted phoneme string generation unit 23 is provided with a function of inserting pseudo phonemes (GB) at locations determined not as caption breaks but as caption sentence breaks. When the phoneme string collation unit 25 calculates the edit distance, the edit cost for GB is set to a value smaller than the edit cost for other phonemes.

  The break of the sentence can be determined by a method of detecting a phrase “.” As a sentence boundary, a method of analyzing a sentence and detecting a sentence boundary, and the like. A method for detecting sentence boundaries by analyzing sentences is described in, for example, Maruyama et al. “Development and Evaluation of Japanese Section Boundary Detection Program CBAP”, Language Processing Society, July 2004 of Natural Language Processing.

Since the number of subtitle characters displayed on one screen is limited, subtitles are not necessarily sent for each sentence. By providing this function to the caption conversion phoneme sequence generation unit 23, it becomes easier to associate a phoneme sequence due to false recognition of noise with a pseudo phoneme inserted in a caption break than a caption conversion phoneme sequence. Even when the text is not sent for each sentence, it is possible to improve the accuracy of the correspondence between the voice and the caption.
(Ninth embodiment)

In the ninth embodiment, the caption converted phoneme string generation unit 23 is provided with a function of inserting pseudophonemes (GB) at both locations determined as caption breaks and caption sentence breaks. When the phoneme string collation unit 25 calculates the edit distance, the edit cost for GB is set to a value smaller than the edit cost for other phonemes. By giving this function to the caption conversion phoneme string generation unit 23, even if there is an error in the determination of the break of the caption text, a pseudo phoneme (GB) is inserted at least for the break of the caption, It is possible to improve the accuracy of correspondence between audio and subtitles.
(Tenth embodiment)

  In the tenth embodiment, pseudo-phonemes (GB) are inserted according to any of the seventh to ninth embodiments, and the speech recognition unit 21 and the speech verification unit 31 perform speech verification when generating the recognition result phoneme sequence. When the section that is considered to correspond to silence in the unit 35 continues for a predetermined time or longer, a function of inserting a pseudo-phoneme (Q) representing silence is provided at a corresponding portion of the recognition result phoneme string. Then, when calculating the editing distance in the phoneme string matching unit 25, the editing cost when replacing the pseudophoneme (Q) representing silence with the pseudophoneme (GB) is set to 0 or a small value, and the pseudophoneme (Q) Is set to a large value in comparison with other editing costs. For example, the editing cost is set as shown below. Note that β1 >> 1, β2 >> 1,0 <α1 <1,0 <α2 <1.

When cph [c] ≠ GB Insertion cost: ins_cost (cph [c]) = always 1
Deletion cost: del_cost (rph [r]) = 1 (when rph [r] ≠ Q)
= β1 (when rph [r] = Q)
Replacement cost: sub_cost (cph [c], rph [r]) = 0 (when cph [c] = rph [r])
= 1
(when cph [c] ≠ rph [r] and rph [r] ≠ Q)
= β2
(When cph [c] ≠ rph [r] and rph [r] = Q)

When cph [c] = GB Insertion cost: ins_cost (cph [c]) = 0
Deletion cost: del_cost (rph [r]) = α1
Replacement cost: sub_cost (cph [c], rph [r]) = 0 (when rph [r] = Q)
= α2 (when rph [r] ≠ Q)

When associating the recognition result phoneme string and the caption converted phoneme string in the phoneme string matching unit 25, the phoneme string at the beginning of a given caption converted phoneme string is incorrectly associated with the phoneme string at the end of the previous utterance. In this case, a large deviation estimation error occurs. However, in the tenth embodiment, since the silent part in the utterance is easily associated with the portion determined as the break of the subtitle or the break of the text of the subtitle, it is possible to improve the accuracy of the association between the audio and the subtitle. .
(Eleventh embodiment)

  In the eleventh embodiment, in the first to tenth embodiments, the process of calculating the edit distance in the phoneme string collating unit 25 is devised to reduce the calculation processing amount of the deviation width estimation. Since the subtitle conversion phoneme sequence in which the phoneme sequence corresponding to each subtitle is concatenated contains many phonemes, it is effective to reduce the amount of calculation processing in collation between the subtitle conversion phoneme sequence and the recognition result phoneme sequence. It is.

  In the process of calculating the editing distance in the phoneme string collating unit 25, first, the first r phoneme of the recognition result phoneme string and the first n phoneme of the phoneme caption conversion phoneme string (n = 1 to C: C are all phoneme numbers of the caption conversion phoneme string) ) And calculate the edit distance between them. Next, the top N items with the smallest value are selected from the edit distances.

  Then, when calculating the edit distance between the first r + 1 phoneme of the recognition result phoneme string and the caption converted phoneme string, it is selected first among the first n phonemes (n = 1 to C) of the caption converted phoneme string. The edit distance is calculated only for those that can be calculated using the top N edit distances, and not for others. The above processing is repeatedly executed to calculate the edit distance between the recognition result phoneme string and the caption converted phoneme string.

  FIG. 11 is an explanatory diagram showing the edit distance calculation process in the eleventh embodiment. As shown in the figure, first, the edit distance between the first r phoneme of the recognition result phoneme string and the first n phoneme (n = 1 to C) of the caption converted phoneme string is calculated. As a result, edit distances d [r, 1], d [r, 2],..., D [r, C] are obtained. Next, the top N d [r, c-1], d [r, c-2],..., D [r, c-N] having a smaller value are selected from these edit distances. Next, when calculating the edit distance between the first r + 1 phoneme of the recognition result phoneme sequence and the subtitle converted phoneme sequence, the edit distance d [r, c-1], d [r, c-2],. , d [r, cN] compute edit distances only for those that can be computed. Here, the computable edit distance is d [r + 1, c-1], d [r + 1, c-2],..., D [r + 1, cN], and d [r + 1, 1], d [r + 1, 2],..., D [r + 1, c-0] can be omitted. The above processing is repeated while sequentially increasing the number of head phonemes in the recognition result phoneme sequence, and calculation is performed up to the edit distance d [R, C].

According to the eleventh embodiment, it is not necessary to calculate a partial editing distance in the combination of the recognition result phoneme string and the caption converted phoneme string, and the shift width can be estimated with a small calculation processing amount.
(Twelfth embodiment)

  In the twelfth embodiment as well, in the first to tenth embodiments, the process of calculating the edit distance in the phoneme string collating unit 25 is devised to reduce the calculation processing amount of the deviation width estimation.

  In the process of calculating the edit distance in the phoneme string collation unit 25, first, the first r phoneme of the recognition result phoneme string and the first n phoneme (n = 1 to C) of the phoneme subtitle conversion phoneme string are collated, and between them Calculate edit distance. Next, the edit distance having the minimum value and the edit distance difference between the edit distance and the minimum edit distance are selected within a predetermined threshold. This selection method is different from the eleventh embodiment.

  Then, when calculating the edit distance between the first r + 1 phoneme of the recognition result phoneme sequence and the phoneme subtitle conversion phoneme sequence, the first n phonemes of the phoneme subtitle conversion phoneme sequence (n = 1 to C), The edit distance is calculated only for those that can be calculated using the selected edit distance and not for others. The above processing is repeatedly executed to calculate the edit distance between the recognition result phoneme string and the caption converted phoneme string.

According to the twelfth embodiment, it is not necessary to calculate a partial editing distance in the combination of the recognition result phoneme string and the caption converted phoneme string, and the shift width can be estimated with a small calculation amount.
(13th Embodiment)

  In the thirteenth embodiment as well, in the first to tenth embodiments, the calculation process of the edit distance in the phoneme string collating unit 25 is devised to reduce the calculation processing amount of the deviation width estimation.

  In the process of calculating the editing distance by the phoneme string collating unit 25, first, the first r phoneme of the recognition result phoneme string is collated with the first n phoneme (n = 1 to C) of the caption conversion phoneme string, and editing between them is performed. Calculate the distance. Next, the first m phoneme having the minimum value is selected from the editing distance.

  Then, when calculating the edit distance between the first r + 1 phoneme of the recognition result phoneme string and the caption converted phoneme string, the first r + 1 phoneme of the caption converted phoneme string and the first mN phoneme of the caption converted phoneme string (N is The edit distance is calculated only for those that can be calculated using the edit distance with the first r phoneme that has already been calculated among the edit distance with the first m + N phoneme from a certain value), and not for the other . The above processing is repeatedly executed to calculate the edit distance between the recognition result phoneme string and the caption converted phoneme string.

  FIG. 12 is an explanatory diagram showing an edit distance calculation process according to the thirteenth embodiment. In the figure, when the first r phoneme of the recognition result phoneme string is matched with the first n phoneme of the caption conversion phoneme string (n = 1 to C), the first r phoneme of the recognition result phoneme string and the first C phoneme of the caption conversion phoneme string The case where the edit distance from -1 to C-5 is calculated is shown. Here, assuming that the editing distance between the first r phoneme of the recognition result phoneme sequence and the first C-2 phoneme of the subtitle conversion phoneme sequence is the minimum, and N = 2 (two phonemes before and after), the first r of the subtitle conversion phoneme sequence When calculating the edit distance between the +1 phoneme and the caption converted phoneme string, the target of the edit distance calculation is the first C-0 to C-4 phonemes of the caption converted phoneme string. However, since the edit distance between the first r phoneme of the recognition result phoneme string and the subtitle converted phoneme string C-0 is not calculated, the edit distance is actually calculated using the first r + 1 phoneme of the subtitle converted phoneme string Subtitle conversion phoneme strings C-1 to C-4 phonemes.

  According to the thirteenth embodiment, it is not necessary to calculate a partial edit distance in the combination of the recognition result phoneme string and the caption converted phoneme string, and the shift width can be estimated with a small calculation amount.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment. In addition, the present invention is a playback device that plays back subtitles corrected for time lag together with audio and video in received broadcast content, and a playback device that searches for and plays back specific video portions and audio portions in broadcast content. Can also be realized. For example, a subtitle that has been corrected for time lag by any of the subtitle shift correction apparatuses of the above embodiments is stored, and a video that matches the input keyword using characters in the stored subtitle Search and play back parts and audio parts. In this case, since the time lag between the video and audio and the subtitle is corrected in view of the information content, a desired video portion and audio portion can be correctly searched from the characters in the subtitle and reproduced.

  11 ... Information separation unit, 12 ... Subtitle start / end time determination unit, 13 ... Deviation correction unit, 21 ... Speech recognition unit, 22 ... Recognition result phoneme string storage unit, 23 ... Subtitle conversion phoneme sequence generation unit, 24 ... Subtitle conversion phoneme sequence storage unit, 25 ... Phoneme sequence collation unit, 26 ... Phoneme sequence collation result storage unit, 31 ... Speech detection unit, 32 ... -Acoustic analysis unit, 33 ... Acoustic model storage unit, 34 ... Language model storage unit, 35 ... Speech collation unit

Claims (20)

A speech recognition unit that recognizes speech in the received broadcast content and generates a recognition result phoneme sequence corresponding to the speech while receiving the broadcast content;
A subtitle conversion phoneme sequence generation unit that generates a phoneme sequence corresponding to each subtitle in the video of the broadcast content and generates a subtitle conversion phoneme sequence by concatenating those phoneme sequences;
Subtitle start and end times by associating subtitles and audio based on the editing distance between the recognition result phoneme sequence generated by the speech recognition unit and the subtitle conversion phoneme sequence generated by the subtitle conversion phoneme sequence generation unit A phoneme sequence matching unit for determining
Based on the start and end times of the subtitles determined by the phoneme string collation unit, a shift correction unit that corrects a time shift between the audio and the subtitles,
The phoneme string matching unit associates the subtitle with the audio before the subtitle reception when receiving the subtitle, and predicts the start and end times of the subtitle after the time lag correction from the result, and then the predicted subtitle The subtitles and audio are associated with each other using the audio and subtitle information in the newly received broadcast content at regular time intervals until the start and end times of the video. A subtitle shift correction apparatus characterized by repeating a process of predicting start and end times.   The phoneme string matching unit is adapted to correct the time lag based on the result of associating the subtitles with the audio before the subtitle reception, the predicted start time of the subtitles, and the subtitle display time acquired when the broadcast content is received. The subtitle shift correction apparatus according to claim 1, wherein an end time of the subtitle is predicted.   The phoneme string collating unit temporally determines a subtitle based on a result of associating a subtitle with audio before receiving the subtitle, a predicted value of the start time of the subtitle, and a predicted value of the audio length corresponding to the subtitle estimated from the subtitle character string. The caption shift correction apparatus according to claim 1, wherein an end time of a caption after shift correction is predicted.   The phoneme string matching unit stores the recognition result phoneme string used when predicting the start and end times of subtitles after correcting the time lag, and the phoneme string matching unit starts the subtitles after correcting the time lag When the process for predicting the end time is repeated, the recognition result phoneme sequence generated by the speech recognition unit from the time when the previous process was performed to the current time is combined with the stored recognition result phoneme sequence. A sequence is generated, and an end time of a caption after time lag correction is predicted based on an edit distance between a recognition result phoneme sequence generated thereby and a caption converted phoneme sequence. 4. The caption shift correction device according to any one of 3 above.   The recognition result phoneme sequence used when the phoneme sequence collation unit predicts the start and end times of the subtitles after the time lag correction, the recognition intermediate result recognized by the speech recognition unit, and the collation intermediate collated by the phoneme sequence collation unit The speech recognition unit saves the result, and the speech recognition unit takes over the recognition intermediate result saved during the previous process and repeats the previous process when the process of predicting the start and end times of the caption after the time lag correction is repeated. The recognition result phoneme string is generated by recognizing the speech from the performed time to the current time and generating a recognition result phoneme string, and the recognition result phoneme string is combined with the stored recognition result phoneme string. 4. The subtitle end time after time offset correction is predicted using the generated recognition result phoneme string and the collation intermediate result saved during the previous processing. Kanji characters Deviation correction device.   The recognition result phoneme sequence used when the phoneme sequence collation unit predicts the start and end times of the subtitles after the time lag correction, the recognition intermediate result recognized by the speech recognition unit, and the collation intermediate collated by the phoneme sequence collation unit The speech recognition unit saves the result, and the speech recognition unit takes over the recognition intermediate result saved during the previous process and repeats the previous process when the process of predicting the start and end times of the caption after the time lag correction is repeated. Recognizing the speech from the performed time to the current time to generate a recognition result phoneme sequence, the phoneme sequence collation unit has recognized the recognition result phoneme sequence and the speech from the time when the recognition was first started to the current time The recognition result phoneme sequence is compared, and the edit distance between the recognition result phoneme sequence and the subtitle conversion phoneme sequence is calculated using different stored phonemes based on the edited distance. And subtitle time gap Subtitles displacement correction device according to any one of claims 1 and estimates 3.   After the shift correction unit corrects the temporal shift width between the audio and the subtitle, the subtitle corresponding to the audio section before the correction time, the recognition result phoneme sequence corresponding to the subtitle, and the subtitle converted phoneme 7. The caption shift correction apparatus according to claim 5, wherein the column and the collation intermediate result are discarded.   After the shift correction unit corrects the temporal shift width between the voice and the subtitle, the recognition result phoneme, the subtitle converted phoneme string, and the collation intermediate result corresponding to the voice section that is a predetermined time before the correction time are discarded. 7. The caption shift correction apparatus according to claim 5 or 6,   The edit distance is defined by using as an index a cost corresponding to the number of steps required to convert from one phoneme string to another by inserting, deleting, or replacing a phoneme. 9. The caption shift correction device according to any one of claims 8 to 10.   The edit distance is determined based on the speech recognition performance for each phoneme, the cost required to replace a phoneme with a phoneme, the cost required to insert a phoneme, and the cost required to delete a phoneme 9. The caption shift correction apparatus according to claim 1, wherein the caption shift correction apparatus is defined as an index.   The subtitle conversion phoneme sequence generation unit generates a subtitle conversion phoneme sequence in which pseudo phonemes representing sentence breaks are added to subtitle breaks, and the phoneme sequence collation unit is smaller than other phonemes for the pseudo phonemes 11. The caption shift correction apparatus according to claim 9, wherein the edit distance is calculated by giving a cost.   The subtitle conversion phoneme sequence generation unit generates a subtitle conversion phoneme sequence in which pseudo phonemes representing sentence breaks are added to sentence breaks obtained by analyzing subtitles, and the phoneme string matching unit The subtitle shift correction apparatus according to claim 9 or 10, wherein the edit distance is calculated with a cost smaller than that of other phonemes.   The subtitle conversion phoneme sequence generation unit generates a subtitle conversion phoneme sequence by adding pseudo phonemes representing sentence breaks to sentence breaks obtained by analyzing subtitle breaks and subtitles, and the phoneme string collation unit includes: 11. The caption shift correction apparatus according to claim 9, wherein the edit distance is calculated by giving a pseudo-phoneme smaller cost than other phonemes.   The subtitle conversion phoneme sequence generation unit generates a subtitle conversion phoneme sequence in which pseudo phonemes representing sentence breaks are added to sentence breaks obtained by analyzing subtitles, and the speech recognition unit continues silence for a certain period of time. A recognition result phoneme string is generated by adding a pseudophoneme representing silence at a location, and the phoneme string collating unit sets the cost between the pseudophoneme representing silence and the pseudophoneme representing a sentence break to 0 or a small value, 11. The caption shift correction device according to claim 9, wherein the edit distance is calculated as a large value by comparing a cost between the pseudo phoneme to be expressed and another phoneme with another. 11.   The phoneme string collating unit calculates an edit distance between the first r phoneme of the recognition result phoneme string and the first n phoneme of the caption converted phoneme string (n = 1 to C: C is the total number of phonemes of the caption converted phoneme string). , When the edit distance between the top r + 1 phoneme of the recognition result phoneme sequence and the subtitle conversion phoneme sequence is calculated, the top N pieces having the smaller value are selected from the edit distances. Of n phonemes (n = 1 to C), the edit distance is calculated only for those that can be calculated using the top N edit distances selected previously, and the other is not repeatedly calculated. 15. The caption shift correcting device according to claim 1, wherein an edit distance between the recognition result phoneme string and the caption converted phoneme string is calculated.   The phoneme string collating unit calculates an edit distance between the first r phoneme of the recognition result phoneme string and the first n phoneme of the caption converted phoneme string (n = 1 to C: C is the total number of phonemes of the caption converted phoneme string). A minimum one of the edit distances and a difference between the edit distance and a minimum difference within a predetermined threshold, and the first r + 1 phoneme of the recognition result phoneme string and the caption converted phoneme string When calculating the edit distance between, the edit distance is calculated only for those that can be calculated using the edit distance selected earlier, among the first n phonemes (n = 1 to C) of the caption conversion phoneme string, 15. The caption shift correcting apparatus according to claim 1, wherein the processing for not calculating the others is repeatedly executed to calculate the edit distance between the recognition result phoneme string and the caption converted phoneme string.   The phoneme string collating unit calculates an edit distance between the first r phoneme of the recognition result phoneme string and the first n phoneme of the caption converted phoneme string (n = 1 to C: C is the total number of phonemes of the caption converted phoneme string). Selecting the first m phoneme of the subtitle conversion phoneme string having the smallest value from the editing distance, and calculating the edit distance between the first r + 1 phoneme of the recognition result phoneme string and the subtitle conversion phoneme string The first r + 1 phoneme of the recognition result phoneme sequence and the first mN phoneme of the caption conversion phoneme sequence (N is a constant value) and the first m + N phoneme already calculated in the editing distance. r Repeat the process of calculating the edit distance only for those that can be calculated using the edit distance with the phoneme and not the others, and calculate the edit distance between the recognition result phoneme string and the subtitle conversion phoneme string. 15. The caption shift correction apparatus according to claim 1. A caption shift correction device according to any one of claims 1 to 17,
A playback device comprising: playback means for playing back a caption whose time shift has been corrected by the caption shift correction device together with audio and video in the received broadcast content. A caption shift correction device according to any one of claims 1 to 17,
Subtitle storage means for storing subtitles corrected for time lag between audio and video by the subtitle shift correction device;
Search means for searching for video of a part that matches the input keyword based on character information in the subtitle stored in the subtitle storage means,
A playback apparatus for playing back the video of the part searched by the search means. A caption shift correction device according to any one of claims 1 to 17,
A broadcasting apparatus comprising: transmission means for transmitting, as a broadcast program, audio, video, and subtitles corrected for temporal deviation by the subtitle deviation correcting apparatus.

Images