JP6849977B2 - Synchronous information generator and method for text display and voice recognition device and method - Google Patents

Description

The present invention relates to a text display synchronization information generation technique and a voice recognition technique, and more specifically to a lyrics synchronization information generation technique for generating synchronization information between lyrics and music.

When playing back acoustic data, you may want to display the text related to this in synchronization with the playing of the acoustic data. For example, when playing music content, the lyrics part may be displayed or highlighted in synchronization with the audio part. There is also a request to display a script (line part) in synchronization with the audio part when playing back the video content.

By the way, if the music data and the lyrics data are not distributed as contents in a state of being associated with each other, the lyrics cannot be displayed in synchronization with the playback of the music data as it is. In such a case, , Synchronous information for lyrics is required separately. The lyrics synchronization format LRC is known as the de facto standard for such synchronization information (https://en.wikipedia.org/wiki/LRC_ (file_format)). FIG. 1 (A) shows a file conforming to the lyrics synchronization format LRC (the lyrics are copyright-free automatically generated by a computer), and the elapsed time (time stamp) from the start of the music signal and the lyrics text. Each display line of is described in association with each other. The time stamp displays, for example, the time at which the corresponding display line begins to be sung. FIG. 1B shows an example of lyrics card text. In order to generate such lyrics synchronization information, the operator performs a predetermined operation at the timing of singing each line while playing and listening to the music, and associates a time stamp with each line, and a synchronization information file, for example, Manual processing such as generating an LRC file was performed. However, in order for the worker to generate the LRC file, it is basically necessary to listen to all the performances of the music, which is complicated. It is desirable to mechanically generate synchronization information without manual intervention.

It is also conceivable to use a general speech recognition engine to generate mechanical synchronization information. However, in general, a singer's singing expresses a pronunciation different from normal conversation depending on each expression style. For example, it refers to an expression that deforms the pronunciation of "I" as a diphthong like "Uwatoasui", or puts a pillow sound like "Akokuro" for "Kokoro". These are the singer's own performances and are present in many songs. In a general speech recognition engine, there is a problem that these expressions are regarded as the sound as it is and are erroneously recognized.

In Patent Document 1, when synchronization information is generated by using a processing device of a small display, each line of the lyrics text (lyric card) is divided into one or a plurality of clauses so as to fit in the size of the small display. It is disclosed that it is divided into display units, and then time stamps are assigned to the display units.

The present invention should not be understood in a limited manner due to the above circumstances or the following problems, and the contents thereof are defined in the claims and will be described in detail below with reference to examples.

Japanese Unexamined Patent Publication No. 2006-227082

The present invention has been made in consideration of the above-mentioned circumstances and the like, and mechanically generates synchronization information or mechanically performs voice recognition, and moreover, the habit of the singer or speaker, the style of acting, and the dialect. The purpose is to suppress problems caused by such factors.

According to the present invention, in order to achieve the above object, the configuration as described in the claims is adopted. Here, prior to explaining the invention in detail, the description of the scope of claims will be supplementarily explained.

According to one aspect of the present invention, in order to achieve the above object, a text display synchronization information generator that determines the timing of displaying each of the partial texts of the text based on the voice signal corresponding to the text. : Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the pronunciation component of the pronunciation feature label for each of a plurality of pronunciation feature labels, and the voice signal is concerned. With the pronunciation feature label score generation means, which generates the likelihood that the segment is the pronunciation component of the pronunciation feature label as a score, and outputs the score of each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment; From the transition of scores for the plurality of pronunciation feature labels generated by the pronunciation symbol score generating means based on the time transition of the voice signal segment, it is possible to individually adapt to a specific speaker or a specific group of speakers, respectively. With reference to a plurality of reference symbol estimation dictionaries, a matching reference symbol string generating means generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment; and a second matching reference symbol string based on the above text. The matching reference symbol string of the above is matched with the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means along a dynamically determined matching path. With a matching means that associates the time information of the voice signal segment with the second matching reference symbol string; the time information of the voice signal segment associated with the second matching reference symbol string is used to describe the above text. It is configured to have a display timing determining means for determining the display timing for displaying each of the partial texts.

According to this configuration, the matching reference symbol string based on the text is matched across a plurality of matching reference symbol strings, so that the matching is successfully performed and the display timing of the partial text is generated. be able to.

In this configuration, the text data may be based on the lyrics text corresponding to the voice signal.

Further, in this configuration, the text data is based on Japanese text, and the matching reference symbol may be kana characters (hiragana or katakana) or romaji notation. In the language of phonograms, phonograms may be widely used as reference symbols for matching.

Further, in this configuration, the text data is based on the English text, and the matching reference symbol may be expressed at the word unit level. It is also applicable to any language that can use word-level units as syntactic elements, such as Spanish, French, German, Portuguese, and Russian.

Further, in this configuration, each matching reference symbol included in the first matching reference symbol string is accompanied by a likelihood with respect to the audio signal, and the matching means R (in order of the magnitude of the likelihood). R may be an integer of 2 or more) as a matching reference symbol.

Further, in this configuration, the phonetic symbol may be a phoneme piece label.

Further, in this configuration, the plurality of reference symbol estimation dictionaries may be adapted by machine learning.

Further, in this configuration, the plurality of reference symbol estimation dictionaries include a general N1 speaker group, a habitual N2 speaker group, and N (= N1 + N2 + N3) for N3 vocalists of individual artists. ) Reference symbol estimation dictionary may be used.

Further, in this configuration, the partial text of the text is a text line, and the display timing determining means determines the display timing for each text line and creates an LRC file that describes the display timing for each text line. You can output it.

Further, according to another aspect of the present invention, text display synchronization information that determines when to display each of the partial texts of the text based on the voice signal corresponding to the text in order to achieve the above object. Generator: A plurality of reference symbol estimation dictionaries that can be individually adapted to a specific speaker or a specific group of voices from each of the voice signal segments obtained by dividing the voice signal along the time axis. With reference to the matching reference symbol string generating means generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment; the second matching reference symbol string based on the above text. Is matched with the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means along a dynamically determined matching path, and the second matching reference is used. A matching means that associates the time information of the voice signal segment with the symbol string; and displays each of the partial texts of the text using the time information of the voice signal segment associated with the second matching reference symbol string. It is configured to have a display timing determining means for determining the display timing to be performed.

According to this configuration, the matching reference symbol string based on the text is matched across a plurality of matching reference symbol strings, so that the matching is successfully performed and the display timing of the partial text is generated. be able to.

Further, according to another aspect of the present invention, text display synchronization information that determines when to display each of the partial texts of the text based on the voice signal corresponding to the text in order to achieve the above object. Generator: Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the phonetic feature label for each of a plurality of phonetic feature labels. A pronunciation feature label score generation means that generates a likelihood that the voice signal segment is a pronunciation component of the pronunciation feature label as a score, and outputs a score for each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment. With reference to the reference symbol estimation dictionary from the transition of the scores for the plurality of pronunciation feature labels generated by the pronunciation symbol score generation means based on the time transition of the voice signal segment, the first matching reference symbol. A matching reference symbol string generating means that generates a column in association with the time information of the voice signal segment; and a second matching reference symbol string based on the text are generated from the matching reference symbol string generating means, respectively. With a matching means that matches the plurality of first matching reference symbol strings along a dynamically determined matching path and associates the time information of the voice signal segment with the second matching reference symbol strings. To have a display timing determining means for determining a display timing for displaying each of the partial texts of the text using the time information of the voice signal segment associated with the second matching reference symbol string. It is configured.

According to this configuration, the matching reference symbol is estimated based on the time transition of the plurality of pronunciation feature label scores to generate the matching reference symbol string, so that the matching reference symbol string based on the voice is not restricted by the language. Can be matched with the matching reference symbol string based on the text, and the display timing of the text part can be generated without being restricted by the language.

Further, in this configuration, the matching reference symbol may be a Japanese kana character.

Further, in this configuration, the matching reference symbol may be expressed at the word unit level.

Further, according to still another aspect of the present invention, the speech recognition device: for each of the speech signal segments obtained by dividing the speech signal along the time axis, the pronunciation feature for each of the plurality of pronunciation feature labels. Filtering is performed so as to emphasize the pronunciation component of the label, the likelihood that the speech signal segment is the pronunciation component of the pronunciation feature label is generated as a score, and each score for a plurality of pronunciation feature labels of the speech signal segment is generated. A pronunciation feature label score generating means that is output in association with time information; a speech recognition unit from a score transition for the plurality of pronunciation feature labels generated by the pronunciation symbol score generating means based on the time transition of the voice signal segment. It is configured to have a voice recognition unit output means that sequentially outputs voice recognition units by referring to an estimation dictionary.

According to this configuration, since the speech recognition unit is estimated based on the time transition of a plurality of phonetic symbol scores, speech recognition can be performed without being restricted by the language type.

Further, in this configuration, the voice recognition unit may be Japanese kana.

Further, in this configuration, the voice recognition unit may be expressed at the word unit level.

It should be noted that the present invention can be realized not only as an apparatus or a system but also as a method. Of course, a part of such an invention can be configured as software. It is also natural that software products (computer programs, computer-readable recording media) used to execute such software on a computer are also included in the technical scope of the present invention.

The above-mentioned aspects and other aspects of the present invention are described in the claims and will be described in detail below with reference to Examples and the like.

According to the present invention, synchronization information that synchronizes a text portion with a voice signal can be mechanically and surely generated, and voice recognition can also be performed.

It is a figure explaining the example of the lyrics synchronization format LRC and the lyrics card text. It is a block diagram explaining the structural example of the synchronous information generation apparatus 1000 of Example 1 of this invention. It is a figure explaining the computer system 300 which mounts the synchronization information generation apparatus 1000 of FIG. It is a flowchart explaining the operation example of the synchronization information generation apparatus 1000 of FIG. It is a figure which shows the example of the text and the reference symbol (hiragana) column generated from this. It is a figure which schematically explains the phonetic symbol space of the phonetic symbol score generation part 1051 of FIG. It is a figure explaining the phonetic symbol extraction filter apparatus 1055 used by the phonetic symbol score generation part 1051 of FIG. It is a figure explaining the operation example of the phonetic symbol extraction filter apparatus 1055 of FIG. FIG. 5 is a diagram schematically illustrating an example of a change amount decibel of a phonetic symbol generated by the phonetic symbol extraction filter device 1055 _{1 of FIG. 7.} It is a figure explaining the output example of the phonetic symbol score. It is a figure explaining the machine learning example of the reference symbol estimation unit 1052 of FIG. It is a figure explaining the output operation example of the reference symbol estimation unit 1052 of FIG. It is a figure explaining the example which selects the 1st reference symbol string by the reference symbol selection part 1053 of FIG. It is a figure explaining the example of matching the 1st reference symbol string selected by the reference symbol selection part 1053 of FIG. 2 with the 2nd reference symbol string. It is a flowchart explaining the operation example of the Example of FIG. It is a flowchart explaining the operation example of the Example of FIG. It is a figure explaining the specific example of synchronous information extraction schematically. It is a figure explaining an example applied to English voice. It is a figure explaining an example applied to English voice. It is a figure explaining an example applied to English voice. It is a block diagram explaining the principle structure example of the synchronous information generation apparatus 200 of Example 2 of this invention. It is a figure explaining the operation example of the synchronization information generation apparatus 200 of FIG. It is a figure explaining the example of the phonetic symbol used in the synchronous information generation apparatus 200 of FIG. It is a figure explaining the katakana probability table of a phonetic symbol. It is a figure explaining the matching of the kana of a text and a phonetic symbol (time stamp). It is a figure explaining Example 3 which applied this invention to voice recognition.

Hereinafter, examples of the present invention will be described. First, the synchronous information generator 1000 according to the first embodiment of the present invention will be described. The synchronization information generation device 1000 generates synchronization information between music data and lyrics text data, and typically generates an LRC format file described with reference to FIG. 1 above. is there.

FIG. 2 shows a principle configuration example of the synchronous information generation device 1000 according to the first embodiment of the present invention. The synchronization information generation device 1000 automatically creates an LRC file by referring to the music sound data and the text data of the lyrics card. The music is based on Japanese, and the vocal voice signal of the music sound data and the text data of the lyrics card are matched at the level of kana characters (hiragana), and as a result of this matching, the text data of the lyrics card and the vocal voice The time information of the signal (musical sound data) is associated with it, and an LRC file is generated based on this association. The symbols used for matching (also referred to as tokens, for example, kana characters) are hereinafter referred to as matching reference symbols or simply reference symbols. The reference symbol string derived from the vocal voice signal is called the first reference symbol string, and the reference symbol string derived from the text data of the lyrics card is called the second reference symbol string.

The synchronization information generator 1000 is not limited to Japanese, and can be applied to various languages. When applied to a language with word breaks such as English, the reference symbol may be, for example, a word that is case-sensitive or abstracts special symbols. In this example, synchronous information generation for lyrics will be described, but the present invention is not limited to this.

In this embodiment, a plurality of first reference symbol strings derived from the vocal audio signal are generated, and the second reference symbol string derived from the text of the lyrics card is cross-matched with the plurality of first reference symbol strings. You can let me. A plurality of first reference symbol generation processes or modules, each of which generates a first reference symbol sequence, are configured by machine learning to be adapted to a specific speaker (speaker) or a specific speaker (speaker) group, respectively. good.

Further, in this embodiment, the voice signal is divided along the time axis to generate a series of voice signal segments (for example, about 0.5 mS), and for each of these voice signal segments, for each of a plurality of phonetic symbols. Filtering is performed so as to emphasize the acoustic component of the phonetic symbol, the likelihood of the phonetic signal segment being the acoustic component of the phonetic symbol is generated as a score, scores are output for a plurality of phonetic symbols, and a set of the scores is obtained. A matching reference symbol, for example, a kana character, is estimated and identified based on the temporal transition of the phonetic symbol score vector or phonetic symbol space information (here, it is also referred to as phonetic symbol score vector or phonetic symbol space information). The phonetic symbol (also called a phonetic feature label) refers to a category of phonetic features, and may be a phoneme unit (vowel, consonant) phonetic symbol as shown later in FIG. 19, and is a phonetic fragment (Demiphoneme) unit phonetic symbol. Is also good. As phonetic symbols for each phoneme piece, "automatic labeling method using phoneme piece network representation of speech and time series segmentation method" (Acoustical Society of Japan, Vol. 42, No. 11 (1986), hereinafter referred to as the Acoustical Society of Japan paper. The phoneme fragment label of) may be used. As the phonetic symbol for each phoneme, a phonetic symbol (including a subset thereof) conforming to the International Phonetic Alphabet may be used.

The sampling period length of the audio signal segment may be fixed or variable. In the case of variable length, a break in the frequency spectrum may be extracted to determine the sampling delimiter. The duration of the audio signal segment is, but is not limited to, about 0.5 mS. The duration of the phoneme is, for example, 60-80 mS, but is not limited to this.

In a specific implementation of this embodiment, as will be described in detail later, a plurality of acoustic filter systems are prepared, individual acoustic filter systems are associated with one phonetic symbol, and the phonetic features of the corresponding phonetic symbols. Is configured to emphasize the input audio signal. Each acoustic filter system produces a corresponding output based on how much the input speech signal contains the phonetic features of the phonetic symbols associated with the acoustic filter system. When the input voice signal corresponds to the phonetic symbol of a predetermined sound filter system, the sound filter system produces the largest output. The set of phonetic symbols emphasized by multiple acoustic filter systems is selected to exactly match the existing set or subset of phonetic symbols, such as the phonetic symbols of the previous Japanese Acoustic Society paper, the phonetic symbols of the International Phonetic Association. It may be slightly deviated depending on the adjustment of the filter device, or it may be completely original. It may be a set of phonetic symbols which will be described later with reference to FIG.

The acoustic filter system is, for example, a resonance filter, and the resonance filter may be configured to include a plurality of notch filter elements.

The output generated by the plurality of acoustic filter systems can grasp the voice characteristic component of the corresponding phonetic symbol as the likelihood contained in the input voice signal. The output can be expressed as, for example, a score of 0 to 1.

The score of multiple phonetic symbols generated for a voice signal segment (q, q is an integer of 2 or more, typically tens or hundreds of orders) is the q dimension of q phonetic symbols. In the space of (hereinafter, these spaces are also referred to as phonetic symbol spaces), the voice characteristics of the voice signal segment are defined. In this embodiment, the reference symbol (hiragana) is determined based on the temporal transition of the speech characteristics of the speech signal segment in such a phonetic symbol space. This will be described later.

FIG. 2 shows the synchronization information generation device 1000 of this embodiment as a whole. In this figure, the synchronization information generation device 1000 includes a music sound data file input unit 10, a lyrics text data file input unit 20, and a vocal voice signal. Extraction unit 30, time information extraction unit 40, first reference symbol string generation unit 1050, display unit line division unit 110, second reference symbol string generation unit 60, reference symbol string matching unit 1070, text matching unit 80, LRC file generation It is configured to include a unit 90, an LRC file storage unit 100, and the like.

In this example, the synchronization information generator 1000 is typically configured by one computer system 300, but may be configured to have a plurality of computer systems and various network systems. A part of the computer system 300 may be composed of an information terminal such as a smartphone. The computer system 300 typically includes a CPU 301, a main memory 302, a bus 303, an auxiliary storage device 304, various input / output interfaces 305, and the like, as shown in FIG. 3, a personal computer, a smartphone, and information. It may be a home appliance or the like. The synchronous information generator 1000 is realized, for example, by installing a computer program recorded on a computer-readable recording medium 306 (FIG. 2) or a computer program transmitted via a communication network (not shown) in the computer system 300. Will be done. The synchronization information file may be stored in the LRC file storage unit 100 and accessible from an external device 400 (FIG. 2) such as a mobile terminal. The external device 400 may upload the music sound data and the lyrics text data to the synchronization information generation device 1000, and acquire the synchronization information file generated by the synchronization information generation device 1000 accordingly. The music sound data file input unit 10 and the lyrics text data file input unit 20 may be configured as an external device.

In FIG. 2, the first reference symbol string generation unit 1050 includes a phonetic symbol score generation unit 1051, a reference symbol estimation unit 1052, and a reference symbol selection unit 1053. The phonetic symbol score generation unit 1051 generates a phonetic symbol score indicating the likelihood that the phonetic signal segment accompanies the phonetic characteristic component of the phonetic symbol for each phonetic symbol, and generates a phonetic symbol score vector having the score of each phonetic symbol as an element. It can be generated and output in association with the time information (time) of the voice signal segment. Reference symbol estimation unit 1052, based on the time series of phonetic symbols score vector that is sequentially output from the phonetic symbol score generation unit 1051, each of the estimated Dictionary 1054 ₁ ~1054 _N of N (N is an integer of 2 or more) With reference to, _{a reference symbol, its score, and time information can be generated for each of the estimation dictionaries 1054 1 to 1054 N} , and a reference symbol string can be output for each of a series of voice signals. Reference symbol estimator 1052 estimates dictionary 1054 _1-1054 with reference to one of the _N well to generate one reference symbol sequence, also with reference of N estimated Dictionary 1054 _{1-1054 N} respectively N You may generate a string of reference symbols. The reference symbol selection unit 1053 preferentially receives one reference symbol string and outputs it as it is (FIG. 13), or receives N reference symbol strings and typically calculates a predetermined evaluation value. As a whole, R (R is a positive integer, where R ≦ N) reference symbol strings are selected and output for a series of audio signals in descending order of likelihood (FIG. 17, step S303). For example, the reference symbol selection unit 1053 is typically estimated by the one estimation dictionary when it is considered that the predetermined one estimation dictionary is appropriately adapted to the speaker, depending on the usage situation. Only one reference symbol string may be output. In this case, it is not necessary to perform the estimation process for other estimation dictionaries. The first reference symbol string is divided into subcolumns of display units or highlight units by the display unit row division unit 110. The reference symbol column matching unit 1070 outputs typically R reference symbol strings, which are output from the first reference symbol string generation unit 150 and divided into subcolumns by the display unit row division unit 110, into the second reference symbol. The second reference symbol string output from the column generation unit 60 is matched so as to select the optimum matching route. The reference symbol string matching unit 1070 divides the second reference symbol string into substrings based on the matching result, and adds time information to these substrings. When there is one first reference symbol string, the time axis may be expanded or contracted for matching. After that, the text matching unit 80 matches the text data with the second reference symbol column (subsequence), and based on the matching result, the time information is associated with the subtext (text line) of the text data. The LRC file generation unit 100 generates an LRC file from text data associated with time information.

In this embodiment, hiragana is used as the reference symbol, but in reality, the reference symbol may be a word uniquely assigned to each utterance unit. As an example of the method of labeling phonetic symbols, the phoneme fragment label of the above-mentioned Acoustical Society of Japan paper may be used. Further, as a machine learning method for handling time-series information, for example, there is RNN (Recurrent Neural Network) for deep learning, but the method is not limited to these. By using a machine-learned model using correct answer data, reference symbol candidates are extracted together with a score value indicating the certainty corresponding to the time-series transition of the input phonetic symbol space information.

In this embodiment, one or more vocal segments are determined as one reference symbol based on the time transition of the phonetic symbol score vector. The time timings of the reference symbol strings for each of N groups are not always the same. For example, when the top seven reference symbol strings are arranged based on the size of the score in a predetermined duration, for example, FIG. 14 is shown.

The reference symbol string matching unit 1070 dynamically matches the reference symbol string (indicated in bold) of the text and the reference symbol sequence (R pieces) of the audio signal in the manner shown in FIG. In this example, the reference symbol string "This time ..." is generated from the text, and the time t-1, t-2, t-3, ..., T- from the first reference symbol string generation unit 1050. Reference symbols are sequentially generated along the line 10. In this example, matching is done along the paths shown in bold.

FIG. 4 illustrates an outline of an operation example of the synchronization information generator 1000 shown in FIG.

In FIGS. 2 and 4, the music sound data file input unit 10 inputs a file in which the sound signal is recorded in a format such as MP3 or MP4 (FIG. 4, step S10). The lyrics text data file input unit 20 inputs a file in which lyrics text (also referred to as lyrics card text) is recorded (FIG. 4, step S11).

The vocal audio signal extraction unit 30 extracts an audio signal (vocal signal, voice signal) from the music signal (FIG. 4, step S12). The vocal voice signal extraction unit 30 supplies the voice signal (vocal signal, voice signal) extracted from the music signal to the time information extraction unit 40. The time information extraction unit 40 sequentially divides the audio signal into audio signal segments having a sampling cycle along the time axis, and imparts time information (time stamp) indicating progress information from the start of the song to each audio signal segment, for example. (FIG. 4, step S13). The time information extraction unit 40 supplies the audio signal segment with the time information to the first reference symbol string generation unit 1050.

As described above, the first reference symbol string generation unit 1050 includes a phonetic symbol score generation unit 1051, a reference symbol estimation unit 1052, and a reference symbol selection unit 1053. The phonetic symbol score generation unit 1051 receives a voice signal segment (vocal signal, voice signal) with time information, and has a predetermined number (q, q is positive) in accordance with the phonetic symbol space described later. For each of the phonetic symbols (integer), the voice signal segment generates a likelihood including the acoustic component of those phonetic symbols as a score, and outputs a q-dimensional score vector in association with time information (FIG. 4, step S114). The phonetic symbol refers to a category of phonetic features and may be a phonetic factor in units of phonemes (Demiphoneme). As the phonetic symbol for each phoneme piece, the phoneme piece label of the above-mentioned Acoustical Society of Japan paper may be used. Phonetic symbols of phoneme units (vowels, consonants) as shown in FIG. 23, which will be referred to later, may be used.

The phonetic symbol score vector with time information from the phonetic symbol score generator 1051 is supplied to the reference symbol estimation unit 1052, and the reference symbol estimation unit 1052 refers to one of the N estimation dictionaries 1054, or Each of the N estimation dictionaries 1054 is sequentially referred to, the reference symbol is estimated according to the time transition of the phonetic symbol score vector, and the estimation reference symbol, the estimated score, and the time information are output (FIG. 4, step S115). .. The reference symbol estimation unit 1052 can output one reference symbol string or N reference symbol strings for a series of audio signals.

When the reference symbol selection unit 1053 outputs only a single reference symbol string (with score and time information) from the reference symbol estimation unit 1052, the reference symbol selection unit 1053 supplies it as it is to the reference symbol string matching unit 1070, and supplies the reference symbol to the reference symbol string matching unit 1070. When N reference symbol strings (with score and time information) are output from the estimation unit 1052, the N reference symbol strings are received and, for example, the score is evaluated over a predetermined group of vocalization periods. Then, R reference symbol strings are selected in descending order of evaluation value and supplied to the reference symbol string matching unit 1070. When the reference symbol selection unit 1053 receives N reference symbol strings, it selects R reference symbols in descending order of score for each audio signal segment and outputs R reference symbol strings. Is also good. The R reference symbol strings output from the first reference symbol string generation unit 1050 (reference symbol selection unit 1053) are as shown in FIG. 14, for example. In the example of FIG. 14, R is 7. For each audio signal segment, R reference symbols may be sorted based on the magnitude of the score. However, when the phonetic symbols of a plurality of continuous voice signal segments are the same, it is preferable to group them together.

The display unit line dividing unit 110 determines the line of the display unit to be displayed or highlighted at one time during sound reproduction. The display unit may be determined based on a desired rule by combining one or a plurality of sections separated by breathing. One specific rule example is, but is not limited to, that the maximum number of characters is a predetermined number, for example, 18 characters. In this embodiment, the first reference symbol string from the first reference symbol column generation unit 1050 is divided into the reference symbol subcolumns for each display unit row by the display unit row division unit 110 (FIG. 4, step S16). The division may be executed by adding a separator (for example, a line feed code). Alternatively, the display unit line dividing unit 110 may be provided so as to divide the lyrics text data or the second reference symbol data in partial line units.

On the other hand, the lyrics text data file input unit 20 inputs text, for example, lyrics text (FIG. 4, step S11). The text is supplied to the second reference symbol string generation unit 60 and the text matching unit 80. The second reference symbol string generation unit 60 generates a reference symbol string (hiragana string. FIG. 5 (B)) from the text data (FIG. 5 (A)) (FIG. 4, step S17). The second reference symbol string from the second reference symbol string generation unit 80 is supplied to the reference symbol string matching unit 1070, and one first reference symbol string from the first reference symbol string generation unit 1050, or R pieces. It is dynamically matched with the first reference symbol string (FIG. 4, step S118). An example of matching the second reference symbol sequence to R first reference symbol sequences is as shown in bold in FIG. By this matching, the time information of the first reference symbol string is given to the second reference symbol string. The second reference symbol string to which the time information is added is supplied to the text matching unit 80.

The text matching unit 80 matches the second reference symbol string with the time information with the text data, and adds the time information to the text data in units of partial lines (FIG. 4, step S19). The LRC file generation unit 90 receives text data to which time information is added in units of partial lines and generates an LRC file (FIG. 1 (A)) (FIG. 4, step S20). This may be stored in the LRC file storage unit 100.

FIG. 6 schematically describes the phonetic symbol space. In the embodiment of FIG. 2, the likelihood of the phonetic signal segments distributed in the phonetic symbol space defined by the set of phonetic symbols is investigated, and the distribution in this phonetic symbol space is continuous voice. It measures how transitions occur in a signal segment, and estimates a reference symbol (also called a phonetic token, a kana character or word) based on the measurement result.

As shown in FIG. 6 (A), pronunciation is accompanied by different pronunciation characteristics depending on the state of vocal organs such as vocal cords, lips, teeth, alveolar, palatine, uvula, tongue, and pharynx. Various pronunciation corrections are performed on the voice signal by acoustic processing (filtering), and the weight (score) for each phonetic symbol is extracted. As schematically shown in FIG. 6B, when each phonetic symbol is emphasized for the example voice signal segment, the amount of change (decibel) of "+4.2" with respect to the phonetic symbol "u" is increased. The obtained amount of change of "+2.6" is obtained with respect to the phonetic symbol "o", and the amount of change is similarly obtained with respect to other phonetic symbols. Similarly, the amount of change can be obtained for the phonetic symbols not shown in FIG. 5 (B). This amount of change indicates how much the original input signal level was changed by emphasizing pronunciation. In this example, the signal level (power) after emphasis is divided by the signal level before emphasis, which is a decibel display, but the present invention is not limited to this. It can be determined that the larger the amount of change (decibel) is, the higher the likelihood is, and this amount of change is converted into a score (likelihood) of 0 to 1 based on the conversion table (the closer to 1 the higher the degree of agreement). It is possible to display what kind of acoustic component the voice signal segment has by the score of the phonetic symbol space. It should be noted that the phonetic symbols other than those with a large score may be ignored and treated as an approximation. Typically, for the score vector, a predetermined number of vector elements may be enabled in descending order of score, and other vector elements with smaller scores may be zero or null.

7 (A) and 7 (B) show a configuration example of the phonetic symbol score generation unit 1051 that generates a phonetic symbol score vector that describes the acoustic features of the phonetic signal segment in accordance with the phonetic symbol space. In FIG. 7 (A) and (B), phonetic symbol score generation unit 1051 includes M number (a positive integer, for example, 64) phonetic symbols extraction filter device 1055 _1, 1055 _2, ..., and 1055 _M .. Each of the phonetic symbol extraction filter devices 1055 _{1 to 1055 M} has P (positive integers, eg 512) acoustic filter systems 1056. The acoustic filter system 1056 is, for example, a resonance filter. In each of the acoustic filter systems 1056, filter adjustments of a plurality of notch filters are set, for example, at three levels to emphasize predetermined phonetic symbols. Each of the P phonetic filter systems 1056 of the phonetic symbol extraction filter device 1055 _{1 to 1055 M} corresponds to each of the phonetic symbols and has the largest filter output when the voice signal of the phonetic feature of the corresponding phonetic symbol is input. Filter adjustments are made to generate values. The inputs and outputs of the acoustic filter system 1056 are fed to the corresponding level detector 1057, which detects the amount of change in power (FIG. 6B, schematically shown in decibels in FIG. 9). Based on this, with reference to the conversion table, the input acoustic signal (audio signal segment) outputs the likelihood of being a pronunciation symbol corresponding to the acoustic filter system 1056 as a score value. The score value from the level detector 1057 is sent to the upper phonetic symbol identification unit 1058, and K (K is a positive integer, K ≦ P) scores are output in descending order based on the size of the score, and the smaller scores are Can be ignored (may be zero or null). From the utterance symbol extraction filter device 1055 _{1 to 1055 M} , (M × K) time, phonetic symbols, and score sets are output. In this way, the phonetic symbol score generation unit 1051 generates and outputs a q-dimensional score vector (including a zero or null element) and time information.

FIG. 8 further describes the phonetic symbol score generation unit 1051. As shown in FIG. 8, the phonetic symbol score generation unit 1051 is managed and set by (P × M) phonetic correction emphasis filter parameters. The phonetic symbol score generator 1051 has M phonetic symbol extraction filter devices 1055, and each of the phonetic symbol extraction filter devices 1055 has P acoustic filter systems 1056 (FIG. 7), and the whole (P × M). ) Filter parameters are set individually. Each of the phonetic symbol extraction filter devices 1055 has P for each of the voice signals (time, voice) segmented along the time t (t-1, t-2, t-3, ...). The score of the corresponding phonetic symbol is generated in association with the time information, and the top K of them are output. FIG. 9 schematically shows decibels of electric power generated by the acoustic filter system 1056 corresponding to each of the phonetic symbols of the phonetic symbol score generation unit 1051. The phonetic symbol extraction filter device 1055 ₁ outputs the scores of (M × K) phonetic symbols selected according to the higher level for each phonetic signal segment.

FIG. 10 shows an output example of the phonetic symbol score output from the phonetic symbol extraction filter device 1055 _{1 to 1055 M (only the one with the highest score).} The utterance symbol may be represented by A [B], where "A" may indicate the acoustic feature of the target sampling period and "B" may indicate the acoustic feature of the next sampling period.

FIG. 11 illustrates machine learning of the estimation dictionary of the reference symbol estimation unit 1052. As described in FIGS. 2 and 4, the reference symbol estimation unit 1052 corresponds to the phonetic symbol spatial information for each phonetic signal segment by the phonetic symbol score generation unit 1051 (phonetic symbol extraction filter device 1055). The output phonetic symbol score vector is subjected to time-series analysis processing in consideration of its continuity, and the corresponding reference symbol is obtained. In this embodiment, in preparation for estimating the reference symbol, the continuity of phonetic symbol spatial information and the relationship with the reference symbol are modeled using a machine learning method. Specifically, as shown in FIG. 11, a pair of voice data and text data to be synchronized, which is known to be a correct combination of information in advance, is input as teacher data (S30, S31), and the machine Learn with the learning module. Actually, the pair of the phonetic symbol score vector (phonetic symbol space information) acquired in step S114 of FIG. 4 and the data obtained by converting the text data into the reference symbol in step S16 of FIG. 4 is used as the teacher data. (S32, S33, S34)). The learned pronunciation space model is stored in the estimation dictionary 1054.

Here, the minimum character unit when speaking a language (hiragana, katakana, etc. in Japanese) is expressed in a plurality of minimum character units such as kanji and symbols spoken with a special meaning. In the case of text containing character strings, it is necessary to use a reference symbol that represents the minimum character unit when speaking a language. As described above, as a machine learning method for handling time-series information, for example, a deep learning RNN (Recurrent Neural Network) may be adopted, but the method is not limited thereto. By using a machine-learned model using correct answer data, candidates for reference symbols corresponding to the time-series features of the input utterance space information are extracted together with score values indicating certainty.

N estimation dictionaries can be generated by machine learning using a plurality of teacher datasets. That is, for the machine learning processing flow shown in FIG. 11, a plurality of trained models are generated by using the teacher data set for each feature group of the singer's singing style. In preparing the teacher data for each group, first, the singers are grouped into N groups according to the difference in the singing characteristics (habits) of the singer. For example, grouping is performed by crouching voices, muffled voices, low specific utterance levels such as "shi" and "hi", and uncommon utterances of specific words such as the accent of southern North America. Examples have a plurality of types as follows.
G1: General group (number of groups n1)
G2: Group with habit (number of groups n2) (singer with characteristic singing style)
G3: Individual artist (number of groups n3)
Total number of groups N = n1 + n2 + n3
Machine learning processing is performed for each group classified in this way, and a trained model for each group is created. A collection of trained models for each of these groups is managed as a reference symbol guessing database (estimation dictionary).

FIG. 12 shows an operation example for unknown voice data. In FIGS. 12 and 2, the trained phonetic symbol space model trained for N groups by the learning process shown in FIG. 11 is used as the estimation dictionary 1054. For each phonetic signal segment, the phonetic symbol score generator 1051 generates phonetic symbol vectors associated with time information (S35, S36), and refers to the learned phonetic symbol space model (estimated dictionary 1054) as reference symbol candidates. And its score are estimated (S37). N reference symbol candidates may be output for each N groups (N estimation dictionaries 1054). As shown in FIG. 14, the reference symbol selection unit 1053 may select a set of N time information, reference symbols, and scores based on, for example, a predetermined evaluation value and output one or R sets. , Not limited to this. The score indicates the certainty that the reference symbol is uttered.

FIG. 13 describes an operation example of the first reference symbol string generation unit 1050, focusing on the reference symbol selection unit 1053. In FIG. 13, the phonetic symbol score generation unit 1051 of the first reference symbol string generation unit 1050 sequentially receives the phonetic signal segments associated with the time information, and receives the phonetic symbol set score (phonetic symbol score vector) and time information. Output sequentially. The reference symbol estimation unit 1052 refers to an estimation dictionary 1054 (FIG. 2) having trained models of groups 1 to N, and generates N reference vector sequences based on the transition of the score of the phonetic symbol set. .. Each reference symbol in the reference symbol sequence has a score. The reference symbol string selection unit 1053 calculates the evaluation values of N reference symbol strings based on the scores of the symbols, and selects and outputs one or a plurality of reference symbol strings whose evaluation values satisfy certain conditions.

FIG. 14 describes an operation example of the reference symbol string matching unit 1070 (FIG. 2). In this example, the reference symbol string “this time ...” is output from the second reference symbol string generation unit 60 (FIG. 2) based on the text. For a series of audio signal segments of time t-1, t-2, t-3, ..., R pieces (this example) from the reference symbol selection unit 1053 (FIG. 2) of the first reference symbol string generation unit 1050. Then, the first reference symbol string of 7) is output as "Otehia ...", "Komuchibiwa ...", ... "Saiyuya ...". The first reference symbol string is arranged based on the size of the evaluation value (score). In the example of the figure, the first reference symbol string having a higher score is arranged on the left side. The reference symbol string matching unit 1070 dynamically matches one second reference symbol string with a plurality of first reference symbol strings along the time axis. Bold indicates the dynamic path selected by matching. Matching costs can be reduced by performing matching in descending order of score (left side in FIG. 14).

By matching of the reference symbol string matching unit 1070, the substring of the second reference symbol is associated with the time information of the audio signal segment. Then, as described above, the text matching unit 80 matches the text data with the second reference symbol column (subsequence), and based on this matching result, the time information is added to the partial text (text line) of the text data. Be associated.

Next, a specific example of synchronizing synchronization information will be described. In this example, as explained earlier, N (= n1 + n2 + n3) reference symbol strings are output, the reference symbol string selection unit 1053 outputs one or R first reference symbol strings, and reference symbol string matching is performed. Unit 1070 matches the first reference symbol string with the second reference symbol string. In this example, if the speaker is already registered as belonging to a specific group of estimation dictionaries (there may be a group with one member), the corresponding estimation dictionary is used in advance. Although the synchronization information is extracted (FIG. 15) and otherwise the synchronization information is extracted using N estimation dictionaries (FIG. 16), this embodiment is not limited to this.

In this synchronization information extraction example, synchronization information is extracted by using a predetermined evaluation value in the flow described with reference to FIGS. 15 and 16. In this example, the calculation function of the evaluation value (evaluation_value) when the total time of voice is t_max is defined as follows, but the calculation function is not limited to this.

evolution_value
= Σnum_basic_symbol (t) + w × Σsum_score (t)
However, (0≤t <t_max, w is a weighting coefficient) .... (Equation 1)
number_basic_symbol (t) is the number of reference symbols in which the score value of the reference symbol is Limit_score_min or more in the time t interval.
Σsum_score (t) is the sum of the scores of the reference symbols in the interval of time t.

If the singer is already registered in any of the groups included in the database for a song whose synchronization information is unknown, the synchronization information is extracted according to the flow shown in FIG. That is, as shown in FIG. 15, the reference symbol sequence is estimated using the trained model of the group corresponding to the singer (S200), and the evaluation value according to (Equation 1) is calculated (S201). If the evaluation value is equal to or greater than Limit_model_score_min1 (first minimum value, operable threshold), synchronization information (LRC file) using reference symbol matching and text matching using reference symbol candidate information from the trained model of the group. (S202, S203, S204, S205). If not, the reference symbol sequence is estimated for the trained models of all groups that have not yet been evaluated (S202, S207), the evaluation values are calculated (S208), and all of the evaluation values are less than Limit_model_score_min1. In that case, the process is stopped (S209, S211). When any of the evaluation values is Limit_model_score_min1 or more, reference signal matching and text matching are performed using the upper R reference symbol examples having high evaluation values, and synchronization information is acquired (S209, S210, S203, S204, S205).

If the singer has not yet been registered in any croup, synchronization information is extracted according to the flow shown in FIG. That is, as shown in FIG. 16, reference symbol strings are estimated using the trained models of all groups, evaluation values are calculated (S300, S301), and all evaluation values are Limit_model_score_min1 (first minimum value). If it is less than the operable threshold value), the process is stopped (S302, S309). If the evaluation value of any of the reference symbol strings is Limit_model_score_min1 (first minimum value) or more, reference signal matching and text matching are performed using the upper R reference symbol examples having the highest evaluation value, and synchronization is performed. Information is acquired (S302, S303, S304, S305, S306). If the highest evaluation value is greater than or equal to the predetermined Limit_model_score_min2 (> Limit_model_score_min1), the singer is registered as belonging to the group with the highest evaluation value (S307, S310). If not, the process ends without registration. In this case, new learning may be performed.

In addition, in FIG. 15 and FIG. 16, as described above, when a group having an evaluation value of Limit_model_score_min1 or more is not found, the processing is stopped with a notification that the processing for the voice is difficult. Further, for the singer whose processing was canceled in step S211 of FIG. 15 or step S309 of FIG. 16, or the singer whose highest evaluation value in the processing in step S309 of FIG. 16 was less than Limit_model_score_min2, if necessary. Prepare the correct answer data of the vocal and the text, create a new learning model according to the flow of FIG. 11, and additionally register the group.

In the flow of FIG. 16, the first reference symbol string for a maximum of N groups is output based on the voice signal (step S115 of FIG. 4). The first reference symbol string is matched with the second reference symbol string (step S17 in FIG. 4) generated from the text to be extracted from the synchronization information (step S118 in FIG. 4). In the example of the matching flow of the reference symbol strings described above with reference to FIG. 14, although the upper R first reference symbol strings are used with reference to the evaluation value, N may be used. In the example of FIG. 14, since they may overlap in the time direction, processing such as not duplicating the same extraction results is performed. The degree of similarity between the first symbol string created in this way and the second reference symbol string is calculated. As a calculation method, cosine similarity and the like can be used. Also, if necessary, the degree of similarity with the character string whose character position is shifted is calculated. Even if 100% matching is not performed, the symbol string having the highest degree of similarity is adopted as the matching result. Since time information is added to the first reference symbol string used for matching in symbol units, time information is added as synchronization information to the second reference symbol string (substring thereof) of the text to be extracted from the synchronization information. Is associated (step S118 in FIG. 4).

When converting the text to be synchronized to the reference symbol, by holding the relationship information between the original text and the converted reference symbol and using that information, the second reference symbol string (its substring) in the previous process. Associate the synchronization time information added to with the original text (partial text). (Step S19 of FIG. 4) As a result, text data with synchronization information can be obtained (step S20 of FIG. 4).

FIG. 17 schematically shows the operation of the embodiment centering on the matching of reference symbols. In FIG. 15, the synchronous symmetric text is "This time, please choose me like this ...". This is converted into a second reference symbol string, "This time I am like this ...". On the other hand, based on the voice signal segments (t-1, t-2, ...), the first reference symbol string of a plurality (7 in this example) is "Ono" from the phonetic symbol score information corresponding to the pronunciation space. "Tehia ...", "Komuchibiwa ...", ... "Saiyuya ..." are generated. The first reference symbol string is arranged based on the size of the evaluation value (score). In the example of the figure, the first reference symbol string having a higher score is arranged on the left side. Along the time axis, one second reference symbol string is dynamically matched with a plurality of first reference symbol strings. Bold indicates the dynamic path selected by matching. The second reference symbol substring "this time" is matched to the first reference symbol substring "this time" as shown in bold, and the time information (time stamp) is added to the second reference symbol substring. It is related, and further related to the corresponding subtext "this time". In this example, a time stamp of time t-1 is added to the beginning of "this time". A time stamp may be added to each character that constitutes the partial text (phrase) "this time". Matching may be completed successfully even if a part of the reference symbol string does not match. For matching, the distance (or similarity) between the individual labels of the reference symbol may be introduced to perform dynamic matching so that the distance is smaller (or the similarity is higher).

In this embodiment, there are the following effects.
Since a sound (pronunciation) and a word (character) corresponding to the sound are associated with each other, a voice dictionary for each language is not required, and processing can be performed using a common pronunciation space model regardless of the language.
The speech recognition rate can be improved by converting the pronunciation learned for each person into words and performing matching with a library (pronunciation space model).

This embodiment is applicable, but not limited to, the following applications.
(1) Extraction of temporal synchronization information between song data in which vocals and accompaniment are mixed for music content and text data of lyrics card (2) Audio data of video data for video content such as drama and movie And extraction of temporal synchronization information with the text data of the dialogue in the scenario (3) Time between the voice data of the voice or video data on the conference and the text data extracted from the voice by the voice recognition engine Extraction of target synchronization information

Next, an example in which Example 1 is applied to English will be described. Although the above embodiment applies to Japanese and uses kana characters as reference symbols, the present invention adopts reference symbols as word-level ones and any language in which word-level units can be used as tokens, Spain. It is also applicable to languages, French, German, Portuguese, Russian, etc.

For lyrics in languages that have ideographic characters (for example, kanji) and phonetic characters (hiragana, katakana, and romaji), such as Japanese, it is preferable to convert them to reference symbols for phonetic characters, but in English In such languages, reference symbols can be constructed on the basis of words. However, in this adaptation example, in order to maintain the uniformity of the explanation of Japanese processing, the English adaptation example uses the one that has been converted to uppercase and lowercase letters and the special characters have been deleted as the reference symbol.

In this English example, the English lyric text is converted into a word-level second reference symbol as shown in FIG. FIG. 19 shows an output example of the first reference symbol string according to the voice signal segment. FIG. 20 shows the overall flow of synchronization information extraction. In this example as well, matching is performed along the bold letters. Synchronization information is added to the partial text.

This is the end of the description of the synchronization information generator 1000 according to the first embodiment of the present invention.

Next, Example 2 of the present invention will be described. FIG. 21 shows a simplified synchronization information generator 200 according to Example 2 of the present invention. In FIG. 21, the parts corresponding to FIG. 2 are designated by the corresponding reference numerals. Similar to the first embodiment, the synchronization information generation device 200 of the second embodiment automatically creates an LRC file by referring to the music sound data and the text data of the lyrics card. The music is based on Japanese, and the vocal voice signal of the music sound data and the text data of the lyrics card are matched at the level of kana characters (hiragana), and as a result of this matching, the text data of the lyrics card and the vocal voice The time information of the signal (musical sound data) is associated with it, and an LRC file is generated based on this association. The symbols used for matching (also referred to as tokens, for example, kana characters) are hereinafter referred to as matching reference symbols or simply reference symbols.

The synchronization information generator 200 is not limited to Japanese, and can be applied to various languages. When applied to English as well, the reference symbol may be, for example, a word that is case-sensitive or abstracts special symbols. In this example, synchronous information generation for lyrics will be described, but the present invention is not limited to this.

In FIG. 21, the synchronization information generation device 200 includes a music sound data file input unit 10, a lyrics text data file input unit 20, a vocal voice signal extraction unit 30, a time information extraction unit 40, a first reference symbol string generation unit 50, and a first. 2. The reference symbol string generation unit 60, the reference symbol string matching unit 70, the text matching unit 80, the LRC file generation unit 90, the LRC file storage unit 100, and the like are included. In this configuration example, the first reference symbol string generation unit 50 includes a phonetic symbol detection unit 51 and a reference symbol estimation unit 52.

In this example, the synchronization information generator 200 is composed of one computer system 300, but may be configured to have a plurality of computer systems and various network systems. A part of the computer system 300 may be composed of an information terminal such as a smartphone. The computer system 300 typically includes a CPU 301, a main memory 302, a bus 303, an auxiliary storage device 304, various input / output interfaces 305, and the like, as shown in FIG. 3, a personal computer, a smartphone, and information. It may be a home appliance or the like. The synchronization information generator 200 is realized, for example, by installing a computer program recorded on a computer-readable recording medium 306 or a computer program transmitted via a communication network (not shown) in the computer system 300. The synchronization information file may be stored in the LRC file storage unit 100 and accessible from an external device 400 such as a mobile terminal. The external device 400 may upload the music sound data and the lyrics text data to the synchronization information generation device 200, and acquire the synchronization information file generated by the synchronization information generation device 200 accordingly.

FIG. 22 shows an outline of an operation example of the synchronization information generation device 200 of FIG. In FIGS. 21 and 22, the music sound data file input unit 10 inputs a file in which an acoustic signal is recorded in a format such as MP3 or MP4 (FIG. 22, step S10). The audio signal (audio) includes a vocal signal (voice signal, voice) and an accompaniment signal in this example. The lyrics text data file input unit 20 inputs a file in which data of lyrics text (also referred to as lyrics card text) is recorded (FIG. 22, step S11). As shown in FIG. 1 (B), the lyrics text is often described so that the block structure of the entire song such as the first chorus and the second chorus of the lyrics in the song can be understood, and the lines are in units that are easy to sing. The list is not broken. In this embodiment, the text may be divided into lines of optimized size (FIG. 22, step S16).

The vocal audio signal extraction unit 30 extracts an audio signal (vocal signal, voice signal) from the music signal (FIG. 22, step S12). Various known methods may be adopted for audio signal extraction. Center canceling and frequency band limitation can be used for stereo vocal extraction. This utilizes the fact that vocal voice is included in both Lch and Rch. Further, for monaural vocal extraction, a frequency region limitation can be used, for example, there is a method of extracting a signal by applying a bandpass filter of a human voice band (for example, 100 Hz to 1000 Hz). In addition, the multiple HPSS method, the binary masking method, and the RPCA method may be adopted.

The vocal voice signal extraction unit 30 supplies the voice signal (vocal signal, voice signal) extracted from the music signal to the time information extraction unit 40. The vocal signal (voice signal) may be analyzed for each group of voices based on the detection of a silent section or the like. The time information extraction unit 40 sequentially divides the audio signal into audio signal segments along the time axis at a predetermined sampling cycle, and for example, time information (time stamp) indicating elapsed information from the start of the song is provided for each audio signal segment. (FIG. 22, step S13). The time information extraction unit 40 supplies the audio signal segment with the time information to the first reference symbol string generation unit 50.

As described above, the first reference symbol string generation unit 50 includes a phonetic symbol detection unit 51 and a reference symbol estimation unit 52. The phonetic symbol detection unit 51 inputs a voice signal segment (vocal signal, voice signal) with time information and outputs the phonetic symbol and time information (FIG. 22, step S14). The phonetic symbols are as shown in FIG. 23, and may be detected by formant analysis, mel frequency cepstrum (MFCC) analysis, or the like. Phonetic symbol data can be output using various digital audio workstations, for example, Pro Tools from Digidesign, USA. The phonetic symbol detection unit 51 sequentially outputs a pair of phonetic symbols and time information (also referred to as a time stamp), which is supplied to the reference symbol estimation unit 52.

The reference symbol estimation unit 52 outputs a reference symbol sequence (hiragana sequence) based on the appearance order of phonetic symbols with reference to the kana probability table 52a (FIG. 22, step S15). The kana probability table 52a represents the probability that an individual time series of phonetic symbols is an individual kana (word). In the kana probability table 52a, as shown in FIG. 24, the probabilities that each of the phonetic symbols, for example, "wg" corresponds to each of the reference symbols (kana), for example, "o", "wa", ... It has been demanded. The reference symbol estimation unit 52 outputs a time stamp paired with the phonetic symbol in association with the katakana string. FIG. 25 shows the result of associating the katakana string with the time stamp. The LRC column of FIG. 25 shows the time (time stamp) at which the utterance symbol is detected when the start point of the music signal is used as the starting point. It is not necessary to assign a time stamp to all the reference symbols (kana), and typically, the time stamp may be matched to the kana centering on the phonetic symbol having a high frequency of appearance, but the time stamp is not limited to this. The first reference symbol string accompanied by the time stamp from the reference symbol estimation unit 52 is divided into display table units by the display unit row division unit 110 and supplied to the reference symbol column matching unit 70.

In this example as well, the display unit line dividing unit 110 may divide the lyrics text data or the second reference symbol data in partial line units instead of dividing the first reference symbol string.

On the other hand, the lyrics text data file input unit 20 inputs text, for example, lyrics text data (FIG. 22, step S11). The text is supplied to the second reference symbol string generation unit 60 and the text matching unit 80. The second reference symbol string generation unit 60 generates a reference symbol string (hiragana string. FIG. 5 (B)) from the text data (FIG. 5 (A)) (FIG. 22, step S17). Conversion from text data to reference symbol strings can be performed by morphological analysis or simple dictionary lookup. The second reference symbol string from the second reference symbol string generation unit 80 is supplied to the reference symbol string matching unit 70 and matched with the first reference symbol string from the first reference symbol string generation unit 50, and as a result, the first reference symbol string is matched. 2 Time information is added to the reference symbol string (FIG. 22, step S18). The second reference symbol string to which the time information is added is supplied to the text matching unit 80.

The text matching unit 80 matches the second reference symbol string with the time information with the text data, and adds the time information to the text data (FIG. 22, step S19). The LRC file generation unit 90 receives text data with time information and generates an LRC file (FIG. 1 (A)) (FIG. 22, step S20). This may be stored in the LRC file storage unit 100.

As shown by the broken line in FIG. 21, a plurality of kana probability tables 52a are provided to generate a plurality of reference symbol strings, and the reference symbol string matching unit 70 has the same dynamic as the reference symbol string matching unit 1070 of FIG. Matching is good.

This is the end of the description of the synchronization information generator 200 shown in FIG. 21 according to the second embodiment of the present invention.

Next, Example 3 in which the present invention is applied to a voice recognition device will be described. FIG. 26 shows the voice recognition device 700 of the third embodiment. In this figure, the voice recognition device 700 includes an input unit 701, a voice signal segment division unit 702, a voice symbol score generation unit 703, and a voice unit estimation unit 704. It includes a voice unit estimation dictionary 705 and an output unit 706. The voice recognition device 700 may also be mounted in a computer system as shown in FIG.

The input unit 701 converts the voice generated by the speaker into a voice signal via a microphone. The audio signal segment division unit 702 divides the audio signal into time intervals and outputs the signal. The phonetic symbol score generation unit 703 outputs a group of phonetic symbols and a set of scores for each phonetic signal segment in accordance with the phonetic symbol space. In this example, the time information is unnecessary because the synchronization information is not acquired. The voice unit estimation unit 704 receives a set of phonetic symbols and scores, and estimates the sound unit by referring to the voice unit estimation dictionary 705. The pronunciation unit is the reference symbol (token) in the example described above, and in the case of Japanese, for example, it may be a hiragana character (mora), and in the case of English, etc., a word (case-sensitive, special character). It may be a discarded one), or it may be a phoneme or the like. When Japanese hiragana characters are output, they may be converted into katakana and kanji as appropriate by referring to the dictionary. The final sentence may be output by referring to the word model and the language model as appropriate.

This is the end of the description of Example 3 of the present invention.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, the present invention can be applied to various uses for synchronizing an audio signal and a text to be displayed. For example, the video content and the dialogue (script) may be synchronized. It is also applicable when the recorded voice (video with voice) and the voice recognition result are synchronized and played back. The present invention is applicable not only to Japanese but also to various languages. It should be noted that a person skilled in the art can make various changes according to the description of the claims and implement various examples, and such examples are also included in the description of the claims.
The technical features described here are listed below.
[Technical feature 1]
In a text display synchronization information generator that determines the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the sound feature label for each of a plurality of sound feature labels, and the voice signal segment is used. Generates the likelihood that is the pronunciation component of the pronunciation feature label as a score, and outputs the score of each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment.
The score transitions for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment can be individually adapted to a specific speaker or a specific speaker group. With reference to a plurality of reference symbol estimation dictionaries, each of the plurality of first matching reference symbol strings is associated with the time information of the voice signal segment, and the matching reference symbol string generating means is generated.
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string,
It is characterized by having a display timing determining means for determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. Synchronous information generator for text display.
[Technical feature 2]
The above text has technical features 1 Symbol placement text display synchronization information generating apparatus based on the lyrics text corresponding to the audio signal.
[Technical feature 3]
The above text is based on the Japanese text, the matching reference symbols, text display synchronization information generating apparatus according to any one of the kana characters of technical features 1 or 2.
[Technical feature 4]
The above text is based on the English text, the text display synchronization information generating apparatus according to any one of the technical features 1 or 2 above matching reference symbol is one which is expressed in the level of the word unit.
[Technical feature 5]
Each of the matching reference symbols included in the first matching reference symbol string has a likelihood for the audio signal, and the matching means R (R is an integer of 2 or more) in the order of the magnitude of the likelihood. ) A synchronous information generator for text display according to any one of technical features 1 to 4 for matching reference symbols for matching.
[Technical feature 6]
The above-mentioned pronunciation feature label is a synchronous information generator for text display according to any one of technical features 1 to 5, which is a phoneme piece label.
[Technical feature 7]
The plurality of reference symbol estimation dictionaries are synchronous information generators for text display according to any one of technical features 1 to 6 adapted by machine learning.
[Technical feature 8]
The above-mentioned plurality of reference symbol estimation dictionaries estimate N (= N1 + N2 + N3) reference symbols for a general N1 speaker group, a habitual N2 speaker group, and N3 speaker of an individual artist. A synchronous information generator for text display according to any one of technical features 1 to 7, which is a dictionary.
[Technical feature 9]
The partial text of the text is a text line, and the display timing determining means determines the display timing for each text line and outputs an LRC file describing the display timing for each text line. Technical feature 1 The synchronization information generator for text display according to any one of 8 to 8.
[Technical feature 10]
In the text display synchronization information generation method for determining the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
The pronunciation feature label score generating means emphasizes the pronunciation component of the pronunciation feature label for each of the plurality of pronunciation feature labels for each of the voice signal segments obtained by dividing the voice signal along the time axis. A step of performing filtering, generating the likelihood that the voice signal segment is the pronunciation component of the pronunciation feature label as a score, and outputting each score for a plurality of pronunciation feature labels in association with the time information of the voice signal segment. ,
The matching reference symbol string generating means is a specific utterer or a specific utterance from the transition of the score for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment. With reference to a plurality of reference symbol estimation dictionaries that can be individually adapted to the group of persons, a step generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment, and
The matching means dynamically determines the second matching reference symbol string based on the text to the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means, respectively. A step of matching along the matching path and associating the time information of the audio signal segment with the second matching reference symbol string,
The display timing determining means includes a step of determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. A method for generating synchronous information for text display, which is characterized by the fact that.
[Technical Features 11]
In the text display synchronization information generation computer program for determining the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
Computer,
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the sound feature label for each of a plurality of sound feature labels, and the voice signal segment is used. Is a pronunciation feature label score generation means, which generates a likelihood that is a pronunciation component of the pronunciation feature label as a score, and outputs a score for each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment.
The score transitions for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment can be individually adapted to a specific speaker or a specific speaker group. A matching reference symbol string generating means, which is generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment by referring to a plurality of reference symbol estimation dictionaries.
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string, and
Used as a display timing determining means for determining the display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. A computer program for generating synchronous information for displaying text.
[Technical feature 12]
In a text display synchronization information generator which has an audio signal input device for inputting an audio signal and a CPU and determines the timing of displaying each of the partial texts of the text based on the audio signal corresponding to the text.
The above CPU
For each of the voice signal segments obtained by dividing the voice signal input from the voice signal input device along the time axis, the pronunciation component of the pronunciation feature label is emphasized for each of the plurality of pronunciation feature labels. Filtering is performed, the likelihood that the voice signal segment is the pronunciation component of the pronunciation feature label is generated as a score, and each score for a plurality of pronunciation feature labels is output in association with the time information of the voice signal segment.
A plurality of reference symbol estimation dictionaries that can be individually adapted to a specific speaker or a specific group of voices from the transition of scores for the plurality of pronunciation feature labels generated based on the time transition of the voice signal segment. With reference to, each of a plurality of first matching reference symbol strings is generated in association with the time information of the voice signal segment.
Matching that is dynamically determined by the second matching reference symbol string based on the text to the plurality of first matching reference symbol strings generated by referring to the plurality of reference symbol estimation dictionaries. Matching along the route, the time information of the audio signal segment is associated with the second matching reference symbol string, and the time information is associated with the second matching reference symbol string.
The second matching reference symbol string based on the text data corresponding to the voice signal is matched with the plurality of the first matching reference symbol strings along the dynamically determined matching path.
Text display synchronization information configured to determine the display timing for displaying each of the partial texts of the text using the time information of the audio signal segment associated with the second matching reference symbol string. Generator.
[Technical Feature 13]
In a text display synchronization information generator that determines the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
From each of the audio signal segments obtained by dividing the audio signal along the time axis, refer to a plurality of reference symbol estimation dictionaries that can be individually adapted to a specific speaker or a specific speaker group. , A matching reference symbol string generating means generated by associating a plurality of first matching reference symbol strings with the time information of the audio signal segment, respectively.
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string,
It is characterized by having a display timing determining means for determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. Synchronous information generator for text display.
[Technical Feature 14]
In a text display synchronization information generator that determines the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the sound feature label for each of a plurality of sound feature labels, and the voice signal segment is used. Generates the likelihood that is the pronunciation component of the pronunciation feature label as a score, and outputs the score of each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment.
From the transition of the scores for the plurality of pronunciation feature labels generated by the pronunciation feature label score generation means based on the time transition of the voice signal segment, the reference symbol estimation dictionary is referred to, and the first matching reference symbol string is used. The matching reference symbol string generating means generated in association with the time information of the voice signal segment,
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string,
It is characterized by having a display timing determining means for determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. Synchronous information generator for text display.
[Technical feature 15]
The above-mentioned reference symbol for matching is a Japanese kana. A synchronous information generator for text display according to technical feature 14.
[Technical features 16]
The matching reference symbol is a text display synchronization information generator according to technical feature 14, which is expressed at the word unit level.
[Technical feature 17]
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the pronunciation component of the pronunciation feature label for each of a plurality of pronunciation feature labels, and the voice signal segment is generated. With a pronunciation feature label score generation means that generates a likelihood that is a pronunciation component of the pronunciation feature label as a score and outputs each score for a plurality of pronunciation feature labels in association with the time information of the voice signal segment;
A voice that sequentially outputs voice recognition units by referring to a voice recognition unit estimation dictionary from the transition of scores for the plurality of phonetic feature labels generated by the phonetic symbol score generation means based on the time transition of the voice signal segment. A voice recognition device comprising a recognition unit output means.
[Technical Feature 18]
The voice recognition unit according to the technical feature 17, wherein the voice recognition unit is Japanese.
[Technical features 19]
The voice recognition device according to technical feature 17, wherein the voice recognition unit is expressed at a word unit level.

1000 Synchronous information generator 10 Music acoustic data file input unit 20 Lyrics text data file input unit 30 Vocal voice signal extraction unit 40 Time information extraction unit 1050 First reference symbol string generation unit 1051 Phonetic symbol score generation unit 1052 Reference symbol estimation unit 1053 Reference symbol selection unit 1054 Estimated dictionary 1055 Phonetic symbol extraction filter device 1056 Acoustic filter system 1057 Level detector 1058 Higher phonetic symbol identification unit 60 Second reference symbol string generation unit 1070 Reference symbol string matching unit 80 Text matching unit 90 LRC file generation unit 100 LRC file storage unit 110 Display unit Line division unit 300 Computer system 301 CPU
302 Main memory 303 Bus 304 Auxiliary storage device 305 Input / output interface 306 Computer-readable recording medium 400 External device 700 Voice recognition device 701 Input unit 702 Voice signal segment division unit 703 Voice symbol score generation unit 704 Voice unit estimation unit 705 Voice unit estimation dictionary 706 output section

Claims (12)

In a text display synchronization information generator that determines the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the sound feature label for each of a plurality of sound feature labels, and the voice signal segment is used. Generates the likelihood that is the pronunciation component of the pronunciation feature label as a score, and outputs the score of each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment.
The score transitions for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment can be individually adapted to a specific speaker or a specific speaker group. With reference to a plurality of reference symbol estimation dictionaries, each of the plurality of first matching reference symbol strings is associated with the time information of the voice signal segment, and the matching reference symbol string generating means is generated.
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string,
It is characterized by having a display timing determining means for determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. Synchronous information generator for text display. The text, the text display synchronization information generating apparatus according to claim 1 Symbol placement based on lyrics text corresponding to the audio signal. The above text is based on the Japanese text, the matching reference symbols, text display synchronization information generating apparatus according to claim 1 or 2 is kana characters. The above text is based on the English text, the text display synchronization information generating apparatus according to any one of claims 1 or 2 above matching reference symbol is one which is expressed in the level of the word unit. Each of the matching reference symbols included in the first matching reference symbol string has a likelihood for the audio signal, and the matching means R (R is an integer of 2 or more) in the order of the magnitude of the likelihood. The synchronization information generation device for text display according to any one of claims 1 to 4, wherein the matching reference symbols are matched. The synchronization information generator for text display according to any one of claims 1 to 5, wherein the pronunciation feature label is a phoneme piece label. The text display synchronous information generator according to any one of claims 1 to 6, wherein the plurality of reference symbol estimation dictionaries are adapted by machine learning. The above-mentioned plurality of reference symbol estimation dictionaries estimate N (= N1 + N2 + N3) reference symbols for a general N1 speaker group, a habitual N2 speaker group, and N3 speaker of an individual artist. The synchronous information generation device for text display according to any one of claims 1 to 7, which is a dictionary. The partial text of the text is a text line, and the display timing determining means determines the display timing for each text line and outputs an LRC file describing the display timing for each text line. The synchronization information generation device for text display according to any one of 8. In the text display synchronization information generation method for determining the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
The pronunciation feature label score generating means emphasizes the pronunciation component of the pronunciation feature label for each of the plurality of pronunciation feature labels for each of the voice signal segments obtained by dividing the voice signal along the time axis. A step of performing filtering, generating the likelihood that the voice signal segment is the pronunciation component of the pronunciation feature label as a score, and outputting each score for a plurality of pronunciation feature labels in association with the time information of the voice signal segment. ,
The matching reference symbol string generating means is a specific utterer or a specific utterance from the transition of the score for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment. With reference to a plurality of reference symbol estimation dictionaries that can be individually adapted to the group of persons, a step generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment, and
The matching means dynamically determines the second matching reference symbol string based on the text to the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means, respectively. A step of matching along the matching path and associating the time information of the audio signal segment with the second matching reference symbol string,
The display timing determining means includes a step of determining a display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. A method for generating synchronous information for text display, which is characterized by the fact that. In the text display synchronization information generation computer program for determining the timing of displaying each of the partial texts of the above text based on the voice signal corresponding to the text.
Computer,
Each of the voice signal segments obtained by dividing the voice signal along the time axis is filtered so as to emphasize the sound component of the sound feature label for each of a plurality of sound feature labels, and the voice signal segment is used. Is a pronunciation feature label score generation means, which generates a likelihood that is a pronunciation component of the pronunciation feature label as a score, and outputs a score for each of the plurality of pronunciation feature labels in association with the time information of the voice signal segment.
The score transitions for the plurality of pronunciation feature labels generated by the pronunciation feature label score generating means based on the time transition of the voice signal segment can be individually adapted to a specific speaker or a specific speaker group. A matching reference symbol string generating means, which is generated by associating a plurality of first matching reference symbol strings with the time information of the voice signal segment by referring to a plurality of reference symbol estimation dictionaries.
The second matching reference symbol string based on the text is used as a matching path dynamically determined for the plurality of first matching reference symbol strings generated from the matching reference symbol string generating means. A matching means that matches along and associates the time information of the audio signal segment with the second matching reference symbol string, and
Used as a display timing determining means for determining the display timing for displaying each of the partial texts of the text by using the time information of the audio signal segment associated with the second matching reference symbol string. A computer program for generating synchronous information for displaying text. In a text display synchronization information generator which has an audio signal input device for inputting an audio signal and a CPU and determines the timing of displaying each of the partial texts of the text based on the audio signal corresponding to the text.
The above CPU
For each of the voice signal segments obtained by dividing the voice signal input from the voice signal input device along the time axis, the pronunciation component of the pronunciation feature label is emphasized for each of the plurality of pronunciation feature labels. Filtering is performed, the likelihood that the voice signal segment is the pronunciation component of the pronunciation feature label is generated as a score, and each score for a plurality of pronunciation feature labels is output in association with the time information of the voice signal segment.
A plurality of reference symbol estimation dictionaries that can be individually adapted to a specific speaker or a specific group of voices from the transition of scores for the plurality of pronunciation feature labels generated based on the time transition of the voice signal segment. With reference to, each of a plurality of first matching reference symbol strings is generated in association with the time information of the voice signal segment.
Matching that is dynamically determined by the second matching reference symbol string based on the text to the plurality of first matching reference symbol strings generated by referring to the plurality of reference symbol estimation dictionaries. Matching along the route, the time information of the audio signal segment is associated with the second matching reference symbol string, and the time information is associated with the second matching reference symbol string.
The second matching reference symbol string based on the text data corresponding to the voice signal is matched with the plurality of the first matching reference symbol strings along the dynamically determined matching path.
Text display synchronization information configured to determine the display timing for displaying each of the partial texts of the text using the time information of the audio signal segment associated with the second matching reference symbol string. Generator.

Images