JP2020012855A - Device and method for generating synchronization information for text display - Google Patents

Abstract

To generate synchronization information for lyric text.SOLUTION: An acoustic data file input section 10, a lyric text file input section 20 and a vocal voice signal extraction section 30 extract a vocal voice signal from an acoustic signal, and a phonetic symbol detection section 40 generates a phonetic symbol and time information from the vocal voice signal. A voice recognition section 50 voice-recognizes the vocal voice signal and outputs a voice recognition result text. A phonetic symbol matching section 60 matches the phonetic symbol with a voice recognition result to make occurrence time information correspond to them. A text matching section 90 matches the voice recognition result text to which the time information is allocated with a lyric text, gives the time information at every text per display unit line and generates a synchronization information file.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a text display synchronization information generation technology, and more specifically, to a lyrics synchronization information generation technology for generating synchronization information between lyrics and music.

  When reproducing sound data, there are cases where it is desired to display a text related thereto in synchronization with the reproduction of the sound data. For example, when reproducing music data, the lyrics part may be displayed or highlighted in synchronization with the audio part. There is also a demand for displaying a script (line) in synchronization with an audio part when reproducing video content.

  By the way, if the music data and the lyrics data are not distributed as contents in a linked state, the lyrics cannot be displayed as it is in synchronization with the reproduction of the music data. , Lyrics synchronization information is required separately. The lyrics synchronization format LRC is known as a de facto standard for such synchronization information (https://en.wikipedia.org/wiki/LRC_(file_format)). FIG. 1A shows a file conforming to the lyrics synchronization format LRC (the lyrics are copyright-free automatically generated by a computer). The elapsed time (time stamp) from the start of the music signal and the lyrics text are shown. Are described in association with each display line. The time stamp indicates, for example, the time at which the corresponding display line starts to be sung. FIG. 1B shows an example of a lyrics card text. In order to generate such lyrics synchronization information, a worker performs a predetermined operation and associates a time stamp with each line at the timing of singing of each line while playing / viewing a song, and associates a time stamp with each line. Manual processing such as generation of an LRC file has been performed. However, in order for the operator to generate the LRC file, it is basically necessary to listen to all the performances of the music, which is complicated. It is desired to generate synchronization information mechanically without manual operation.

  According to Patent Document 1, when synchronizing information is generated using a processing device for a small display, each line of lyrics text (lyric card) is separated from one or more phrases so as to fit in the size of the small display. It discloses that the display unit is divided into display units and a time stamp is assigned to the display unit.

  It should be noted that the present invention should not be understood in a limited manner due to the above circumstances or the following problems, the contents of which are defined in the claims, and will be described in detail below using examples.

JP 2006-227082 A

  The present invention has been made in view of the above circumstances and the like, and has as its object to generate synchronization information mechanically.

  According to the present invention, in order to achieve the above-described object, the configuration as described in the claims is adopted. Here, before describing the present invention in detail, the description of the claims will be supplementarily described.

  According to one aspect of the present invention, in order to achieve the above-described object, a lyric synchronization information generation device: discriminates a voice feature from a voice signal and outputs a voice feature symbol together with time information corresponding to a utterance time. Voice recognition means for voice-recognizing the voice signal and outputting a text; and outputting the voice feature symbol output from the phonetic symbol detection means and the text output from the voice recognition means in the order of appearance. First matching means for matching the text along with the text output from the speech recognition means in the order of appearance; and a plurality of texts output from the speech recognition means. Text division means for dividing the text into line portion texts; a matching result of the first matching means; And a display timing determining unit that determines a display timing of each of the plurality of line-portion texts by associating the time information with each of the plurality of line-portion texts based on the result of the switching. .

  According to this configuration, the accurate time information associated with the speech feature symbol is associated with the line portion text by matching the speech feature symbol with the speech recognition result and matching the speech recognition result with the text. The display timing (time stamp) of the partial text can be automatically and reliably generated.

  In this configuration, an audio signal extracting means for extracting the audio signal from the music signal may be further provided. Musical pieces often include singing voices (vocals) and accompaniment, and it is necessary to extract audio signals (here, also referred to as singing voice signals and vocal signals) from the music signals.

  Any method can be used as a method of extracting a voice signal (vocal signal) from a music signal. As a general method, for example, center cancellation and frequency band limitation can be used for stereo vocal extraction. This utilizes that the vocal sound is included in both the Lch and the Rch. For monaural vocal extraction, frequency region restriction can be used. For example, there is a method of extracting a signal through a band-pass filter of a human voice band (for example, 100 Hz to 1000 Hz). In addition, as other techniques with high separation accuracy of the audio portion, a multiplex HPSS method, a binary masking method, an RPCA method, or the like is used.

In this configuration, the audio feature detecting means may output a time from the start of the music signal as the time information. Fine adjustment (time delay or advance) of time information may be set and output.

  In this configuration, the voice feature detection means may output a phonetic symbol as a voice feature symbol. Phonetic symbols can be extracted by formant analysis, mel frequency kepultrum (MFCC) analysis, or the like.

  In this configuration, the voice recognition unit may divide the voice signal at a portion where the output of the voice signal is small, and perform voice recognition in units of divided portions.

  In this configuration, the audio feature detecting means may determine a typical audio feature appearing in the audio signal, and correct the audio signal so that the typical audio feature is easily detected.

  For example, pre-processing is performed on a vocal signal extracted from a song signal containing a song to improve the detection accuracy of a speech signal (articulation analysis signal). That is, after extracting an utterance signal from the vocal signal, the utterance signal having a high appearance frequency is checked, the waveform is corrected so that the utterance signal is easily detected, and then the utterance signal is re-executed by the utterance signal detection engine. Signal detection accuracy can be improved, and erroneous detection can be reduced. As the waveform correction processing (pronunciation correction processing example), audio data is decomposed by a number of band-pass filters, and the harmonic composition (amount for each band) is modulated to “a”, “i”, “u”, “e”, “o”. Although the correction of the vowel sound signal such as vowels may be performed, the present invention is not limited to this method.

  Further, in this configuration, the first matching unit converts a speech feature symbol corresponding to the typical speech feature detected corresponding to the typical speech feature into a text output from the speech recognition unit. And the display timing determination means uses the time information assigned to the voice feature symbol corresponding to the typical voice feature detected in correspondence with the typical voice feature, and The display timing of each of the line portion texts may be determined.

  In this configuration, the second matching means may perform matching based on the edit distance corresponding to the lyrics text and the text output from the speech recognition means, but employs various other matching methods. Good.

  In this configuration, the second matching unit performs matching based on an edit distance between the translation of the lyrics text into a different language and the translation of the text output from the speech recognition unit into the different language. good. The application language may be Japanese, but is not limited to this. Any language may be used as long as it can perform voice recognition, display text, and detect voice characteristics. That is, the voice recognition means outputs the Japanese text by voice-recognizing Japanese, and the second mapping means converts the Japanese lyric text and the Japanese text output from the voice recognition means. Although matching may be performed, the present invention is not limited to this and may be applied to various languages. When the application language is Japanese, the different language described above may be English, but is not limited thereto.

  In this configuration, the text division unit may divide the text into the plurality of line portion texts based on an interval at which the audio signal becomes substantially zero.

  In this configuration, the text division unit may divide the partial text based on an allowable range of the number of characters of the partial text. The number of characters is, for example, 18 at the maximum. Further, a division rule based on the structure of the lyrics data may be learned. The learning information can be accumulated every time the synchronization information is generated, and the synchronization information file may be updated according to a new division rule.

  Further, in this configuration, a display timing interpolating unit that performs comparison between a plurality of chorus portions in the lyrics text and interpolates the display timing of the line portion text for which time information could not be obtained based on the comparison result may be further provided. .

  Further, in this configuration, when there is an additional text included in the speech recognition result of the speech recognition unit and not included in the lyrics text, a line portion text is additionally generated based on the additional text, and the additionally generated line portion text is generated. May be assigned a display timing.

  According to another aspect of the present invention, a text display synchronization signal generating device includes: a voice feature detecting unit that determines a voice feature from a voice signal and outputs a voice feature symbol together with time information corresponding to a utterance time; Voice recognition means for voice-recognizing the voice signal and outputting text; a first feature for matching the voice feature symbol output from the phonetic symbol detection means and the text output from the voice recognition means in the order of appearance. Matching means; second matching means for matching the display target text and the text output from the voice recognition means in the order of appearance; and dividing the text output from the voice recognition means into a plurality of line portion texts Text dividing means; and a matching result of the first matching means and a matching result of the second matching means. Zui and, the time information in association with each of the plurality of row sub text, are configured to have a display timing determining means for determining the display timing of each of the plurality of row sub text. If the text output from the voice recognition unit is appropriately divided, the text division unit may be omitted.

  According to this configuration, the accurate time information associated with the speech feature symbol is associated with the line portion text by matching the speech feature symbol with the speech recognition result and matching the speech recognition result with the text. The display timing (time stamp) of the partial text can be automatically and reliably generated.

  The display target text may be a text of a line of video content or a lyrics text of music content.

  According to another aspect of the present invention, a text display synchronization information generating apparatus includes: a voice feature detection unit that determines a voice feature from a voice signal and outputs a voice feature symbol together with time information corresponding to a utterance time. Voice recognition means for voice-recognizing the voice signal and outputting a text; matching for matching the voice feature symbol output from the phonetic symbol detection means and the text output from the voice recognition means in the order of appearance. Means for dividing the text output from the speech recognition means into a plurality of line part texts; and associating the time information with each of the plurality of line part texts based on a matching result of the matching means. And display timing determining means for determining a display timing of each of the plurality of line portion texts. There. If the text output from the voice recognition unit is appropriately divided, the text division unit may be omitted.

  According to this configuration, the corresponding text can be displayed in synchronization with the audio signal.

  The present invention can be realized not only as a device or a system, but also as a method. In addition, it goes without saying that a part of such an invention can be configured as software. Also, it goes without saying that a software product (computer program) used for causing a computer to execute such software is also included in the technical scope of the present invention.

  The above and other aspects of the invention are set forth in the following claims, and are described in detail below using examples and the like.

  According to the present invention, synchronization information for synchronizing a text portion with an audio signal can be mechanically and reliably generated.

It is a figure explaining the example of lyrics synchronous format LRC and lyrics card text. FIG. 1 is a block diagram illustrating a configuration example of a synchronization information generation device according to an embodiment of the present invention. It is a figure explaining the general aspect of lyrics synchronization. FIG. 5 is a diagram illustrating an operation example of the embodiment. FIG. 5 is a diagram illustrating a part of FIG. 4 in detail. It is a figure explaining a phonetic symbol. FIG. 3 is a diagram illustrating an example of voice recognition. It is a figure explaining the Kana probability table of a phonetic symbol. FIG. 5 is a diagram illustrating matching between text kana and phonetic symbols (time stamps). It is a figure explaining adjustment of a line feed position in consideration of a breathing point. It is a figure explaining the correspondence of the text of the speech recognition result and the text of the lyrics card. It is a figure explaining the text matching between a speech recognition result and a lyrics card text. FIG. 11 is a diagram illustrating generation of an initial edit distance when matching is performed using an edit distance by translating into English. FIG. 9 is a diagram for generating an edit distance when a part of a text is changed. FIG. 9 is a diagram for generating an edit distance when a part of a text is changed. FIG. 9 is a diagram for generating an edit distance when a part of a text is changed. FIG. 11 is a diagram illustrating a state in which the edit distance has disappeared due to a partial change of text. It is a figure explaining text matching (decision of a substitution candidate) between a Japanese speech recognition result and a lyrics card text. It is a figure explaining the example of composition of a lyrics chorus. It is a figure explaining the example which extracts information for interpolation from lyrics card data. It is a figure explaining the example which interpolates time information using the information for interpolation from lyrics card data. It is a figure explaining the example which interpolates the vocal voice part which is not in a lyrics card text. It is a figure explaining the aspect which interpolates the vocal sound part which is not in a lyrics card text. It is a figure explaining a breathing point together with lyrics and a waveform. It is a figure explaining the example of extraction of the section without a vocal. FIG. 4 is a diagram illustrating a part of an LRC file output according to an embodiment.

  Hereinafter, a synchronization information generating apparatus according to an embodiment of the present invention will be described. This synchronization information generation device generates synchronization information between music data and lyrics text data, and typically generates the LRC format file described above with reference to FIG. 1 as an example. It is.

  FIG. 2 shows a synchronization information generation device 200 of this embodiment. In this figure, the synchronization information generation device 200 includes an audio data file input unit 10, a lyrics text file input unit 20, a vocal audio signal extraction unit 30, Phonetic symbol detection unit 40, speech recognition unit 50, phonetic symbol matching unit 60, vocal silence section detection unit 70, display unit line division unit 80, text matching unit 90, display unit line synchronization information determination unit 100, lyrics block analysis unit 110 , An interpolation unit 120, a synchronization information file output unit 130, and the like. The type of output information is appropriately described in each block. “Voice” is a vocal audio signal (voice signal), “phs” is a phonetic symbol, “text” is text, “time” is time information (time stamp), and “LRC” is synchronization information. The synchronization information generation device 200 is typically configured using one or a plurality of computer resources (computer systems). An example of the operation of the embodiment of FIG. 2 is described in FIG. 4 and will be referred to hereafter as appropriate. In this example, the generation of lyrics synchronization information will be described, but the present invention is not limited to this. In this example, although it is configured by one computer system 300, it may be configured to include a plurality of computer systems or various network systems. A part of the computer system may be constituted by an information terminal such as a smartphone. The synchronization information generation device 200 is realized by, for example, installing a computer program recorded on the recording medium 301 or a computer program transmitted via a communication network (not shown) in the computer system 300. The computer system includes a CPU, a main memory, a bus, an external memory, various input / output interfaces, and the like, and may be a personal computer, a smartphone, an information home appliance, or the like. The synchronization information file may be stored in the synchronization information file storage unit 140 and be accessible from an external device 400 such as a mobile terminal. The external device 400 may upload the audio data and the lyrics text data to the synchronization information generation device 200, and acquire the synchronization information file generated by the synchronization information generation device 200 in response thereto.

As shown in FIG. 3, (A) character synchronization, (B) line synchronization, and (C) block synchronization are known as modes for displaying lyrics in synchronization with vocals. In this embodiment, description will be made at the level of (B). It is difficult to realize automation in the order of (A), (B), and (C), and it is necessary for an operator to directly input.
(A) Character synchronization: A method of synchronizing time at the position of each character in the lyrics.
(B) Line synchronization: A method of synchronizing the time at the head position of the text line of each lyrics, and additionally approximately adjusting the time of the lyrics in the line.
(C) A method of synchronizing only the time of the beginning position of the chorus (No. 1, 2 etc.) of each lyrics and roughly adjusting the other times.

  Return to FIG. The sound data file input unit 10 inputs a file in which sound signals are recorded in a format such as MP3 or MP4 (FIG. 4, S10). The audio signal (audio) includes a vocal signal (audio signal, voice) and an accompaniment signal in this example. The lyrics text file input unit 20 inputs a file in which lyrics text (also referred to as lyrics card text) is recorded (FIG. 4, S11). For example, as shown in FIG. 1B, the lyrics text is often described so that the block configuration of the entire song such as the first chorus and the second chorus of the lyrics in the song can be understood. There is no line break. In this embodiment, as will be described later, the display unit rows are optimized by the display unit row dividing unit 80 in units that are easy to sing.

  The vocal audio signal extraction unit 30 extracts an audio signal (vocal signal, voice signal) from the music signal (FIG. 4, S12). Various known methods may be adopted for audio signal extraction. Center canceling and frequency band limitation can be used for stereo vocal extraction. This utilizes that the vocal sound is included in both the Lch and the Rch. For monaural vocal extraction, frequency region restriction can be used. For example, there is a method of extracting a signal through a band-pass filter of a human voice band (for example, 100 Hz to 1000 Hz). In addition, a multiplex HPSS method, a binary masking method, and an RPCA method may be adopted.

  The vocal sound signal extraction unit 30 supplies a sound signal (vocal signal, voice signal) extracted from the music signal to the phonetic symbol detection unit 40, the voice recognition unit 50, and the vocal silence section detection unit 70.

  The phonetic symbol detection section 40 receives a voice signal (vocal signal, voice signal) extracted from the music signal and outputs phonetic symbols and time information (FIG. 4, S13). The phonetic symbols are as shown in FIG. 6 and may be detected by formant analysis, mel frequency cepstrum (MFCC) analysis, or the like. A variety of digital audio workstations can be used to output phonetic symbol data, for example, Digidesign's Protocols, USA. In this embodiment, a phonetic symbol having a high frequency of appearance in a voice signal is examined, a waveform correction process is performed so that the phonetic signal is accurately detected, and then the detection engine of the phonetic symbol detection unit 40 of the phonetic signal again. Call back. As a result, the detection accuracy of the sound signal is improved, and erroneous detection can be reduced. By improving the detection accuracy of the utterance pattern using this processing, the matching accuracy with the subsequent text is also improved. As an example of pronunciation correction processing, voice data is decomposed by a number of band-pass filters, and the harmonic composition (amount for each band) is modulated to produce vowels such as “A”, “I”, “U”, “E”, “O” Signal correction may be performed, but is not limited to this.

  An operation example of the phonetic symbol detection unit 40 is shown in FIG. 5 together with an operation example of speech recognition described later. In the operation example of FIG. 5, phonetic symbols and time information are extracted by using a voice feature extraction engine (S131), and detected phonetic symbols are grouped in units of similar phonetic symbols (cluster, S132), and the appearance frequency Is corrected so that the waveform of the phonetic symbol group having a high phonetic symbol approximates the waveform of a typical phonetic symbol (S133), and realization time information is newly extracted from the corrected voice signal using the voice feature extraction engine. (S134).

  The pair of phonetic symbols and time information (also referred to as a time stamp) are sequentially output from the phonetic symbol detection unit 40 in FIG. 2 and supplied to the phonetic symbol matching unit 60.

The voice recognition unit 50 receives a voice signal (vocal signal, voice signal) extracted from the music signal, performs voice recognition, and outputs a Japanese character string (FIG. 4, S14). The speech recognition unit 50 may include not only a speech recognition function as a speech recognition engine, but also functions such as morphological analysis, syntax analysis, and semantic analysis. The speech recognition unit 50 recognizes the speech signal and outputs Japanese text data. The output data includes, for example, a sequence of kana (syllables) and is supplied to the phonetic symbol matching unit 60.
In this example, as a realization example of the voice recognition unit 50, a character string is converted into a Japanese character string and matching with kana (syllables) is performed. However, the language is not specified in Japanese, and lyrics in other languages are used. It is apparent that the matching can be performed by the same processing.

  If the speech recognition engine of the speech recognition unit 50 tends to be weak in recognizing continuous speech signals, the speech signal is divided into appropriate lengths at zero crossing points (positions where the signal level is small), and divided in units of division. Preferably, a voice recognition process is performed. FIG. 7 shows such pre-processing, in which alternating thinning is performed, two thinned audio signals (thinning 1 and thinning 2) are formed from the original audio signal, and speech recognition is performed separately. After that, a speech recognition result is synthesized as post-processing. If necessary, the waveform may be subjected to filter processing so that a specific word (kana) can be easily recognized. A specific operation example is shown in FIG. That is, in the operation example of FIG. 5, the thinning of the audio signal (generation of a plurality of patterns) is performed using the delimiter of the zero crossing point (S141), and the conversion to the Japanese text by the voice recognition engine is performed (S142). The conversion result of the thinning pattern is synthesized (S143).

  The phonetic symbol matching unit 60 matches the phonetic symbols to the kana sequence of the speech recognition result with reference to the kana probability table 60a (step S15 in FIGS. 4 and 5). The kana probability table 60a represents the probability that, when an individual phonetic symbol appears, it is an individual kana (word). It may be the conditional probability of the preceding symbol. In the kana probability table 60a, as shown in FIG. 8, the probabilities that each of the phonetic symbols, for example, "wg" corresponds to each of the words (kana), for example, "wo", "wa", ... are obtained. ing. The phonetic symbol matching unit 60 matches the phonetic symbol to the kana sequence of the speech recognition result, and as a result, outputs a time stamp paired with the phonetic symbol in association with the kana sequence. FIG. 9 shows a result of the association between the kana sequence and the time stamp. The LRC column in FIG. 9 indicates the time (time stamp) at which the utterance symbol was detected starting from the start point of the music signal. Note that, although a time stamp is typically matched to kana with a focus on pronunciation symbols having a high appearance frequency, the present invention is not limited to this.

  The phonetic symbol matching unit 60 assigns a time stamp to the kana of the text of the speech recognition result and outputs the kana by matching (FIG. 9). The text with time information of the phonetic symbol matching unit 60 is sent to the display unit line dividing unit 80 and further supplied to the text matching unit 90.

  The vocal silence section detection unit 70 detects a section in which a voice signal is not basically generated, for example, a prelude, an interlude, a postlude, and a breath part (FIG. 4, S16). The voice silence section is detected when the moving average of the vocal voice signal amplitude or the like falls below a threshold as shown in FIG. A prelude, an interlude, a postlude, and a breath are detected from the length of the section, the occurrence timing, and the like. FIG. 25 shows an example of detecting a section of breath after a prelude. In the example of FIG. 25, lyrics texts are also shown in association with each other. The vocal silent section detection section 70 outputs a silent section and its type.

  The display unit line dividing unit 80 determines a line of a display unit to be displayed or highlighted at a time during sound reproduction (FIG. 4, S18). The display unit line dividing section 80 may determine one or more display unit lines based on a desired rule by combining one or more sections separated by breathing. One rule may be that the maximum number of characters is 18 characters. A display unit line may be determined from various parameters using existing lyrics card data and an example of a display unit break (including the result generated in this embodiment). The display unit line may be described by a line feed code or the like, but is not limited to this. The model learning unit 80a learns a line feed position model, and the display unit line division unit 80 may refer to this. FIG. 10 shows an example of division by the display unit line division unit 80. In this example, the 27-character text is divided into 9-, 5-, 9-, and 6-character display unit lines in accordance with breathing.

  The text matching unit 90 receives the voice recognition text data with time information output from the phonetic symbol matching unit 60, further receives the lyrics text data input from the lyrics text file input unit 20, and matches them (FIG. 4, S19). The matching can be anything that matches one text to the other. In this example, the matching is performed so that the editing distance (difference information) is equal to or less than a predetermined threshold, but the present invention is not limited to this.

  In this embodiment, the matching between the speech recognition text and the lyrics text is performed by evaluating the editing distance, but after editing (partially changing) in Japanese, it is translated in a different language (also called an intermediate language), in this example, translated in English. To evaluate the editing distance between the translations. In another example, the editing distance may be evaluated in Japanese.

First, an example in which the editing distance is indirectly evaluated through translation will be described. In this example, (1) differences in notations such as kanji and katakana are absorbed between texts to be matched.
(2) The edit distance is evaluated in consideration of the meaning of the Japanese word.
(3) Text matching other than Japanese is also performed.
This has the effect.

  In order to evaluate the edit distance indirectly via translation, Japanese is replaced with an intermediate language suitable for difference extraction using the same conversion (translation) engine. In this embodiment, the intermediate language is English, and a UNIX (trademark) diff command is used as the difference extraction engine. After translating Japanese into English, a difference is extracted using the diff command.

  Here, as shown in the example of FIG. 11, it is assumed that there is a text (with time information) of a speech recognition result and a lyrics card text.

  FIG. 12 shows an operation example of indirectly obtaining the edit distance in the English translation. As shown in FIG. 12, first, a partial cutout is performed from the speech recognition result, and a sentence is cut out from the lyrics card text (S191). . The range in which partial cutout (a unit for performing matching) is performed may be one chorus (a block divided by a prelude, an interlude, a postlude, or the like) or a unit smaller than that. Although the speech recognition result text of FIG. 11 is cut out by the chorus size (separated by a silence section of a prelude, an interlude, and a subsequent), the present invention is not limited to this.

  The extracted speech recognition result text and lyrics card text are translated into an intermediate language, in this example, English (S191). Both texts are translated by the same translation engine. Next, difference information is formed using the difference information detection engine for the text of the speech recognition result and the translation result of the lyrics card text (S192), and the editing distance is obtained (S193). At this stage, the text of the original speech recognition result and the lyrics card text (original, no word replacement) are targeted. The speech recognition result and the translation result of the lyrics card text are as shown in FIG. 13A, and the comparison between the texts is shown in FIG. As shown in FIG. 13 (C), the editing distance is 12, as shown in FIG. 13 (D). In the example of FIG. 13D, the Levenshtein distance is used, but the present invention is not limited to this. In FIG. 13C, “difference information”, for example, “5, 8c5, 6” is the fifth to eighth words of the left text (file1) and the fifth to sixth words of the right text (file2). Is changed (c), and “42” of “the number of changes in file1 and file2” indicates that the number of changes in the left text (file1) is 4 and the number of changes in the right text (file2) is 2 Indicates that there is. The edit distance takes the larger of the number of changes in file1 and the number of changes in file2.

In FIG. 13B, “<” indicates a changed portion of the voice recognition text, and “>” indicates a changed portion of the lyrics card text. In FIGS. 13B, 13C, and 13D, the difference information indicates a position at which there is a difference between File1 and File2. The symbol a means addition, d means deletion, and c means change. The line number of file1 is displayed before the symbol, and the line number of file2 is displayed after the symbol. If there are differences over multiple lines, the difference start line and difference end line are displayed separated by commas (,). I do.

  Next, it is determined whether or not the editing distance is equal to or less than a predetermined threshold (S194). If the value is equal to or less than the threshold, the text of the speech recognition result and the lyrics card text successfully correspond to each other, and the lyrics card text is cut out for each display unit line of the speech recognition results, and the cut out portion is displayed in the lyrics card text display unit. A line is written to the LRC file (see FIG. 1), and a time stamp of the first kana of the display unit line of the speech recognition result is written to the LRC file as a time stamp of the display unit line (S197). Thereafter, the next section is matched, and if there is no matching section, the process ends (S199). Note that the time stamp may be advanced or delayed by a predetermined adjustment time.

  If the editing distance is larger than the predetermined threshold, the process proceeds to step S195, and it is determined whether or not the matching has failed. For example, even if rewriting is performed a predetermined number of times (see step S196), the editing distance is predetermined. If the threshold value is exceeded, the matching section is notified that matching has failed (S198). Thereafter, the next section is matched, and if there is no matching section, the process ends (S199).

  If the matching failure condition is not satisfied in step S195, the process proceeds to step 176, where a part of one text is changed with a part of the other text. In this example, a part of the text of the speech recognition result is changed with a part of the lyrics card text, but the reverse is also possible. After changing the text of the speech recognition result, the process proceeds to step S191, where difference information is formed using the difference information detection engine for the translation result of the text (changed) of the speech recognition result and the lyrics card text (original) (S192). , The edit distance is obtained (S193). In this example, as shown in FIG. 14A, the speech recognition result “Neyaya” was changed to “Sister”, and “Osato” was changed to “Sato”. The text of the speech recognition result after the change is as shown in FIG. 14 (B), and the translation result is as shown in FIG. 14 (C). Since there is no change in the lyrics card text, the translation result remains as shown in FIG. The difference information between the translation of the speech recognition result after the change and the translation of the lyrics card text is as shown in FIG. 14 (D), and the editing distance is 9 as shown in FIG. 14 (E). If the edit distance exceeds the predetermined threshold in step S194, the process proceeds to steps S195 and S196 to repeat the processing. If the edit distance is equal to or smaller than the predetermined threshold value of step S197 in step S194, the process proceeds to step S197, and a matching result is output.

  Here, it is assumed that the current edit distance 9 is larger than the threshold value of step S194, and the process proceeds to step 196 again via step S195.

  FIG. 15 shows an example in which the text of the speech recognition result is further changed. In this example, "Neyaya" was changed to "Sister", "Osato" was changed to "Sato", then "Yuki" was changed to "Go", and "Letter" was changed to "News". (FIG. 15A). The order of priority and the unit in which the change is made may be appropriately selected. It may be the order in the text or grammatical information (nominal, declinable, etc.). The number of changes may be fixed or variable.

  In this case, the text of the speech recognition result after the change is as shown in FIG. 15B, and the translation result is as shown in FIG. 15C. Since there is no change in the lyrics card text, the translation result remains as shown in FIG. The difference information between the translation of the speech recognition result after the change and the translation of the lyrics card text is as shown in FIG. 15D, and the editing distance is 5 as shown in FIG. 15E. If this editing distance exceeds the predetermined threshold value of step S194, the process proceeds to step S196 and the process is repeated. If the editing distance is equal to or smaller than the predetermined threshold value in step S194, the process proceeds to step S197, and a matching result is output.

  FIG. 16 shows an example in which the text of the speech recognition result is changed in still another mode. In this example, the previous "Neya" was changed to "Sister", "Osato" was changed to "Sato", then "Yuki" was changed to "Go", and "Endure" was changed to "Endure". It is changed to "End of life" (FIG. 16A).

  In this case, the text of the speech recognition result after the change is as shown in FIG. 16B, and the translation result is as shown in FIG. 16C. Since there is no change in the lyrics card text, the translation result remains as shown in FIG. The difference information between the translation of the speech recognition result after the change and the translation of the lyrics card text is as shown in FIG. 16D, and the editing distance is 5 as shown in FIG. 16E. If this editing distance exceeds the predetermined threshold value of step S194, the process proceeds to step S196 and the process is repeated. If the editing distance is equal to or smaller than the predetermined threshold value in step S194, the process proceeds to step S197, and a matching result is output.

  FIG. 17 shows an example in which the text of the speech recognition result is changed in still another mode. In this example, "Neya" is changed to "Sister", "Yuki" is changed to "Go", "Osato" is changed to "Sato", "Letter" is changed to "News", "Endure" Is changed to "Zetsuhate" (FIG. 17A). The text of the voice recognition result after the change is as shown in FIG. 17B, which is the same as the lyrics card text. Therefore, the editing distance between the translated speech recognition result and the lyrics card text is zero.

  As described above, when the editing distance between translations is equal to or less than a predetermined value, it is determined that the matching is successful, and the display unit line of the lyrics card and its time stamp are output in the matching section. . The part where the matching has failed is notified of the failure. The part where the matching has failed may be interpolated using the lyrics block analysis unit 110 (FIG. 2).

  In the above-described example, the text matching unit 90 outputs the difference information using the translation into the intermediate language and outputs the edit distance. However, the text matching unit 90 directly converts the Japanese text without performing the conversion into the intermediate language. The difference information may be output, and the editing distance may be output.

  FIG. 18 shows an example of outputting difference information and edit distance using Japanese text. In the example of FIG. 18, FIG. 18A shows a text as a result of speech recognition, and FIG. 18B shows a word string obtained by performing word division by morphological analysis. FIG. 18C shows a lyrics card text, and FIG. 18D shows a word string obtained by performing word division by morphological analysis. The text comparison between the text of the speech recognition result and the lyrics card text, each of which is word-delimited, is as shown in FIG. 18 (E), and the difference information is as shown in FIG. 18 (F). Is as shown in FIG. The editing distance is output from the difference information (the number of changed words in the speech recognition text is 7, and the number of changed words in the lyrics card text is 4, so the editing distance is 7 in this example), and is equal to or less than a predetermined threshold value If so, the matching is successful.

  Note that the word replacement candidate in FIG. 18 can be used to change part of the text in step S176 in FIG.

  The display unit line synchronization information determination unit 100 of FIG. 2 is based on the display unit line determined by the display unit line division unit 80 and the matching result (time stamp) of the display unit line based on the matching result executed by the text matching unit 90. Is determined (FIG. 4, S19). The time stamp of the display unit line is basically a time stamp associated with the first kana of the recognition result text portion assigned to the display unit line, but is not limited to this.

  The lyrics block analysis unit 110 of FIG. 2 receives the lyrics text from the lyrics text file input unit 20 and analyzes the lyrics block (FIG. 4, S20). FIG. 20 shows the result of information extraction from the lyrics card data. First, as shown in FIG. 20 (A), the lyrics block number, chorus number, lyrics text, Hepburn romanized notation, prelude, interlude, and subsequent are extracted. Next, as shown in FIG. 20B, the similarity between the lyrics blocks is evaluated based on, for example, the Levenshtein distance, and a pair of lyrics blocks having a small distance is determined. The similarity may be calculated by various methods, for example, a cosine distance may be used. In this example, the distance between the blocks is as shown in FIG. When the Levenshtein distance is equal to or less than a predetermined threshold value and it can be determined that the lyrics blocks are similar to each other, the time information is diverted from another successfully matched lyrics block.

  When there is a lyrics block that can be diverted, the interpolation unit 120 calculates and adds a time stamp corresponding to the display unit line corresponding to the lyrics block (S21 in FIG. 4). In this example, if time information cannot be obtained for the second chorus (the third and fourth lines), as shown in FIG. 21, the first chorus (the first and second lines) or the fourth chorus The time information may be interpolated from the chorus (the seventh and eighth lines).

  If there is a repetition part omitted from the lyrics card, the interpolation unit 120 may interpolate the text and time information of the display unit line using the speech recognition text. In the example shown in FIG. 22, the repetition of “Hey Hey Girl” and “La La La La La La” is omitted in the lyrics card text (or added by vocals), so the text and time information of the repetition part are interpolated. Write to LRC file. An example of this operation is shown in FIG.

  The data output from the interpolation unit 120 is output from the synchronization information file output unit 130 as an LRC file. An output example is as shown in FIG. 26, for example. An LRC file that differs depending on the screen display size of the device that displays the text may be included, and the text may be displayed in a different manner depending on the device. Fine adjustment of display start time (delay or advance of display) may be set for each device.

  This is the end of the description of the embodiment.

  The present invention is not limited to the above-described embodiment, but can be variously modified without departing from the gist of the invention. For example, the present invention can be applied to various uses for synchronizing an audio signal and a text to be displayed. For example, video content and dialogue (script) may be synchronized. The present invention is also applicable to a case where the recorded voice (video with voice) and the voice recognition result are reproduced in synchronization with each other. The present invention is applicable not only to Japanese but also to various languages. Also, it should be noted that those skilled in the art can make various changes and implement various embodiments according to the description of the claims, and such embodiments are also included in the description of the claims.

Reference Signs List 10 Acoustic data file input unit 20 Lyric text file input unit 30 Vocal voice signal extraction unit 40 Phonetic symbol detection unit 50 Voice recognition unit 60 Phonetic symbol matching unit 60a Kana probability table 70 Vocal silent section detection unit 80 Display unit line division unit 80a Model Learning unit 90 Text matching unit 100 Display unit line synchronization information determination unit 110 Lyric block analysis unit 120 Interpolation unit 130 Synchronization information file output unit 140 Synchronization information file storage unit 200 Synchronization information generation device 300 Computer system 301 Recording medium 400 External device

According to one aspect of the present invention, in order to achieve the above object, a lyric synchronization information generating apparatus includes: determining a voice feature from a voice signal, and determining a voice feature symbol and time information of an appearance time of the voice feature. a voice feature detecting means for outputting pairs sequentially in; speech recognition means for the speech signal is recognized voice to output the voice recognition text and; the speech features output from the detecting means the above speech features symbol and the voice recognition means has been the above speech recognition text matching along the order of appearance in the speech recognition text the time first associating information matching unit and an output from; the speech recognition text output from the speech recognition means multiple text dividing means for dividing the row partial text; lyrics text and, said plurality of row sub text in the text dividing means are outputted from said speech recognition means And it divided the speech recognition text matching along the order of appearance, a second matching in correspondence with each of the plurality of row sub text of the speech recognition text retrieve multiple row sub text from the lyric text means and; on Symbol the time information associated with the speech recognition text by the first matching means, retrieved in association with each of the plurality of row sub text of the speech recognition text by said second matching means Further, a display timing determining means for determining a display timing of each of the plurality of line portion texts of the lyrics text in association with each of the plurality of line portion texts of the lyrics text is configured.

In this configuration, the second matching unit performs matching based on an edit distance between the translation of the lyrics text into a different language and the translation of the text output from the speech recognition unit into the different language. good. The application language may be Japanese, but is not limited to this. Any language may be used as long as it can perform voice recognition, display text, and detect voice characteristics. That is, the speech recognition unit outputs the Japanese text by speech recognition of Japanese, and the second matching unit compares the Japanese lyrics text with the Japanese text output from the speech recognition unit. Although matching may be performed, the present invention is not limited to this and may be applied to various languages. When the application language is Japanese, the different language described above may be English, but is not limited thereto.

According to another aspect of the present invention, a text display synchronization signal generating device: discriminates a speech feature from a speech signal, and sequentially outputs a pair of a speech feature symbol and time information of the appearance time of the speech feature. a voice feature detecting unit; the speech recognition output from the speech features the speech feature symbols output from the detection means and the speech recognition means; speech recognition means and for outputting a speech recognition text of the speech signal and speech recognition text dividing the speech recognition text output from the speech recognition means into a plurality of row sub text; first matching means and the matching along the text appearance order associate the time information to the speech recognition text dividing means; display object text, after being divided into the plurality of row sub text in the text dividing means are outputted from said speech recognition means Matching along the order of appearance and the speech recognition text, a second matching means for extracting a plurality of row sub text from the display target text in association with each of the plurality of row sub text of the speech recognition text; the time information associated with the speech recognition text by the upper Symbol first matching means, retrieved in association with each of the plurality of row sub text of the speech recognition text by said second matching means, the Display timing determining means for determining a display timing of each of the plurality of line portion texts of the display target text in association with each of the plurality of line portion texts of the display target text . If the text output from the voice recognition unit is appropriately divided, the text division unit may be omitted.

Further, according to another aspect of the present invention, a text display synchronization information generating apparatus includes: determining a speech feature from a speech signal, and sequentially pairing a speech feature symbol and time information of the appearance time of the speech feature. a voice feature detecting unit for outputting; the speech signal a speech recognition means for outputting a speech recognition text by speech recognition and, the output from the speech features symbol and the voice recognition means is output from the voice feature detecting means text segmentation for dividing the speech recognition text output from the speech recognition means into a plurality of row sub text; matching means and a speech recognition text matching along the order of appearance associate the time information to the speech recognition text means and; by the matching means by using the time information associated with the speech recognition text, on the speech recognition text It is configured to have a display timing determining means for determining the display timing of each of the plurality of row sub text. If the text output from the voice recognition unit is appropriately divided, the text division unit may be omitted.

An operation example of the phonetic symbol detection unit 40 is shown in FIG. 5 together with an operation example of speech recognition described later. In the operation example of FIG. 5, phonetic symbols and appearance time information are extracted by using a speech feature extraction engine (S131), and detected phonetic symbols are grouped in units of similar phonetic symbols (cluster, S132). A group of frequent phonetic symbols is corrected so as to have a waveform similar to a typical phonetic symbol (S133), and appearance time information is extracted again from the corrected voice signal using a voice feature extraction engine. (S134).

The phonetic symbol detection unit 40 of FIG. 2 sequentially outputs pairs of phonetic symbols and appearance time information (also referred to as time stamps), which are supplied to a phonetic symbol matching unit 60.

The text matching unit 90 receives the speech recognition text data with time information divided into display lines output from the phonetic symbol matching unit 60, and further receives the lyrics text data input from the lyrics text file input unit 20. The two are matched (FIG. 4, S19). The matching can be anything that matches one text to the other. In this example, the matching is performed so that the editing distance (difference information) is equal to or less than a predetermined threshold, but the present invention is not limited to this.

Claims (24)

Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
First matching means for matching the speech feature symbol output from the phonetic symbol detection means and the text output from the speech recognition means in the order of appearance;
Second matching means for matching the lyrics text and the text output from the voice recognition means in the order of appearance;
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts,
The time information is associated with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit, and each of the plurality of line portion texts is associated with the time information. A lyrics synchronization information generating apparatus, comprising: display timing determining means for determining display timing.   2. The lyrics synchronization information generating apparatus according to claim 1, further comprising audio signal extracting means for extracting the audio signal from a music signal.   3. The lyrics synchronizing information generating apparatus according to claim 2, wherein the audio feature detecting means outputs a time from a start point of the music signal as the time information.   4. The synchronizing information generator for lyrics according to claim 1, wherein said voice feature detecting means outputs pronunciation symbols as voice feature symbols.   5. The lyrics synchronizing information generating apparatus according to claim 1, wherein the voice recognition means divides the voice signal in a portion where the output of the voice signal is small and performs voice recognition in units of divided portions.   The said audio | voice characteristic detection means determines the typical audio | voice characteristic which appears in the said audio | voice signal, and corrects the said audio | voice signal so that the said typical audio | voice characteristic is easy to be detected. Lyric synchronization information generator.   The first matching means matches a speech feature symbol detected corresponding to the typical speech feature and corresponding to the typical speech feature with a text output from the speech recognition means, and The timing determining means uses the time information assigned to the speech feature symbol corresponding to the typical speech feature detected in correspondence with the typical speech feature, to extract the plurality of line portion texts. 7. The lyrics synchronizing information generating device according to claim 6, wherein each display timing is determined.   8. The lyrics synchronization information generating apparatus according to claim 1, wherein said second matching means performs matching based on an edit distance corresponding to the lyrics text and the text output from the voice recognition means.   9. The method according to claim 8, wherein the second matching unit performs matching based on an edit distance between the translation of the lyrics text into a different language and the translation of the text output from the speech recognition unit into the different language. Lyric synchronization information generator.   The speech recognition means outputs a Japanese text by speech recognition of Japanese, and the second mapping means matches the Japanese lyrics text with the Japanese text output from the speech recognition means. The lyrics synchronization information generating apparatus according to claim 9.   11. The lyrics synchronization information generating apparatus according to claim 10, wherein the different language is English.   The lyrics synchronizing information generating apparatus according to any one of claims 1 to 11, wherein the text dividing means divides the text into a plurality of the line portion texts based on an interval at which the audio signal becomes substantially zero.   13. The lyrics synchronizing information generating apparatus according to claim 12, wherein the text dividing unit divides the partial text based on an allowable range of the number of characters of the partial text.   14. A display timing interpolating means for comparing lyrics texts between a plurality of chorus parts and interpolating display timing of a line part text for which time information could not be obtained based on the comparison result. The synchronizing information generator for lyrics described in 1.   If there is an additional text included in the voice recognition result of the voice recognition means and not included in the lyrics text, a line portion text is additionally generated based on the additional text, and a display timing is set for each of the additionally generated line portion texts. 15. The lyrics synchronizing information generating apparatus according to claim 1, wherein the synchronizing information is assigned. Voice feature detecting means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice recognition of the voice signal and output text,
A first matching unit that matches the voice feature symbol output from the phonetic symbol detection unit with the text output from the voice recognition unit in the order of appearance;
A second matching unit that matches the lyrics text with the text output from the voice recognition unit in the order of appearance;
Text dividing means for dividing the text output from the voice recognition means into a plurality of line portion texts,
A display timing determining unit that associates the time information with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit; Determining a display timing of each of the line part texts. Computer
Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
First matching means for matching the speech feature symbol output from the phonetic symbol detection means and the text output from the speech recognition means in the order of appearance;
Second matching means for matching the lyrics text with the text output from the voice recognition means in the order of appearance;
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts, and
The time information is associated with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit, and each of the plurality of line portion texts is associated with the time information. A computer program for generating synchronization information for lyrics used to be realized as display timing determining means for determining display timing. Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
First matching means for matching the speech feature symbol output from the phonetic symbol detection means and the text output from the speech recognition means in the order of appearance;
Second matching means for matching the display target text and the text output from the voice recognition means in the order of appearance;
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts,
The time information is associated with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit, and each of the plurality of line portion texts is associated with the time information. A synchronizing information generation device for text display, comprising: a display timing determining means for determining a display timing.   19. The text display synchronization information generating apparatus according to claim 18, wherein the display target text is a text of a line of video content. Voice feature detecting means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice recognition of the voice signal and output text,
A first matching unit that matches the voice feature symbol output from the phonetic symbol detection unit with the text output from the voice recognition unit in the order of appearance;
A second matching unit that matches the display target text and the text output from the voice recognition unit in the order of appearance;
Text dividing means for dividing the text output from the voice recognition means into a plurality of line portion texts,
A display timing determining unit that associates the time information with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit; Determining a display timing of each of the line portion texts. Computer
Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
First matching means for matching the speech feature symbol output from the phonetic symbol detection means and the text output from the speech recognition means in the order of appearance;
Second matching means for matching the display target text and the text output from the voice recognition means in the order of appearance,
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts, and
The time information is associated with each of the plurality of line portion texts based on the matching result of the first matching unit and the matching result of the second matching unit, and each of the plurality of line portion texts is associated with the time information. A computer program for generating synchronous information for text display used to be realized as a display timing determining means for determining a display timing. Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
A matching unit that matches the speech feature symbol output from the phonetic symbol detection unit and the text output from the speech recognition unit in the order of appearance;
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts,
Display timing determining means for associating the time information with each of the plurality of line-portion texts based on the matching result of the matching means and determining the display timing of each of the plurality of line-portion texts. Synchronization information generation device for text display. Voice feature detecting means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice recognition of the voice signal and output text,
Matching means for matching the voice feature symbol output from the phonetic symbol detection means and the text output from the voice recognition means in the order of appearance,
Text dividing means for dividing the text output from the voice recognition means into a plurality of line portion texts,
Determining a display timing of each of the plurality of line portion texts by associating the time information with each of the plurality of line portion texts based on a matching result of the matching portion. Synchronization information generation method for text display, characterized in that: Computer
Voice feature detection means for determining a voice feature from the voice signal and outputting a voice feature symbol together with time information corresponding to the utterance time;
Voice recognition means for voice-recognizing the voice signal and outputting a text,
Matching means for matching the speech feature symbol output from the phonetic symbol detection means and the text output from the speech recognition means in the order of appearance,
Text division means for dividing the text output from the speech recognition means into a plurality of line portion texts, and
The time information is associated with each of the plurality of line-portion texts based on the matching result of the matching unit, and is used as a display-timing determination unit that determines a display timing of each of the plurality of line-portion texts. Computer program for generating synchronization information for displaying text.