JP2019020597A - End-to-end japanese voice recognition model learning device and program - Google Patents

Abstract

To improve a Japanese voice recognition rate.SOLUTION: An end-to-end Japanese voice recognition model learning device 2 comprises: label generation means 20 for generating a class label which allocates a plurality of characters whose appearance frequency included in a text is lower than previously determined appearance frequency reference to a class representing the plurality of characters from a text 1b in learning data 1, and a single label on a single character whose appearance frequency is higher than the reference; text generation means 5 for converting the character included in the text 1b into the class label on the basis of a character/label conversion table 4 for allocating he character to the class label, and generating a converted text 1c being a text after the text is converted; and acoustic model learning means 6 for learning the voice 1a, the converted text 1c, the class label and the plurality of single labels, converting the voice 1a into a label string of the class label and the single label by learning, and learning the acoustic model on the basis of the converted label string.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an end-to-end Japanese speech recognition model learning device and a program thereof.

Conventionally, learning of a speech recognition model for recognizing speech generally requires the following processes (1) to (3).
(1) Learning an acoustic model that converts input speech into feature vectors (acoustic feature quantities), (2) converting feature vectors into phonemes assigned manually to each word, and (3) phonemes Learn a language model that converts columns into words.

  Among them, the process (2) of the prior art first learns an acoustic model by HMM / GMM (Hidden Markov model / Gaussian Mixture Model), which is resistant to changes in the sequence length of the input speech, and more recently, DNN (Deep Neural Network). ) To generate a high-accuracy acoustic model by learning the acoustic model (see FIG. 12A).

  As a technique for eliminating the complexity of such model learning, an acoustic model learning method using CTC (Connectionist Temporal Classification) (see Non-Patent Document 1) and DNN is known. This learning method is a mechanism for directly learning the correspondence between speech and labels such as phonemes or characters. This learning method is strong in the learning ability of the acoustic model even when the sequence length of the input speech is changed. By replacing the learning by HMM / GMM with the acoustic model learning method using CTC and DNN, Learning can be performed collectively (End-to-End). In particular, various methods are known for the acoustic model learning method using CTC and RNN (Recurrent Neural Network). By using a large amount of data called big data, the feature vector of the input speech is input, A learning method for directly outputting characters (character labels) from this feature vector has also been proposed (see Non-Patent Documents 2 and 3). In end-to-end acoustic model learning, as shown in FIG. 12B, intermediate representations such as phonemes are not used.

Graves, A., et al., “Connectionist Temporal Classification: Labeling Unsegmented Sequence Data with Recurrent Neural Networks,” ICML '06 Proceedings of the 23rd international conference on Machine learning Pages 369-376 (2006) Miao, Y., et al., ”EESEN: END-TO-END SPEECH RECOGNITION USING DEEP RNN MODELS AND WFST-BASED DECODING” 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) Pages 167-174 (2015) Hannun, A., et al., “Deep Speech: Scaling up end-to-end speech recognition” Cornell University Library arXiv: 1412.5567, 19 Dec 2014

However, many of the previous studies that use DNN output as characters are intended for speech recognition in English. When Japanese is handled, the following two problems arise due to the large number of Japanese character types.
One is that the number of output labels is large and the number of parameters is enormous compared to English. When an end-to-end neural network (NN) that outputs characters in English is configured, the number of output labels is about 100, including numbers and symbols in the alphabet, but in Japanese, kanji, hiragana, katakana There are more than 3,000 character types. There are many coupling parameters between each layer of the network due to the large number of character types, but since there are not many types of pronunciations for the character types, duplication occurs in the representation in the network, and the model is not robust.
Make learning difficult.

  Another problem is the problem of data sparsity, which means so-called “sparseness”. In the case of Japanese, the number of average learning samples per character decreases as the number of character types increases, and there are also characters with extremely low appearance frequencies. In an acoustic model in which characters with a low appearance frequency (low frequency characters) exist in the output label, the acoustic characteristics of the characters are hardly learned, and as a result of speech recognition, for example, unnecessary low frequency characters are erroneously inserted. Tended to be output as. For this reason, it has been difficult to improve the speech recognition rate.

  The present invention has been made in view of the above problems, and an object thereof is to provide an end-to-end Japanese speech recognition model learning apparatus and program capable of improving the speech recognition rate in Japanese. To do.

In order to solve the above-mentioned problem, an end-to-end Japanese speech recognition model learning device according to the present invention is a learning data including text and speech or an acoustic feature amount of the speech. An end-to-end Japanese speech recognition model learning device that learns an end-to-end acoustic model that outputs a character or word label from a speech recognition model,
From the text in the learning data, a plurality of characters or a plurality of words whose appearance frequency included in the text is lower than a predetermined appearance frequency standard are assigned to a class representing the plurality of characters or the plurality of words. Label creating means for creating a class label and a single label attached to a single character or single word having a higher appearance frequency than the reference;
Based on the conversion table that assigns the plurality of characters or the plurality of words to the class label, the plurality of characters or the plurality of words included in the text in the learning data are converted into the class label, and the text is converted. A text creation means for creating a post-conversion text that is a later text;
The speech that is the learning data or the acoustic feature amount of the speech, the converted text, the class label, and the plurality of single labels are learned, and the speech or acoustic feature amount is learned by the learning and the class label and An acoustic model learning unit that converts the label string into the single label and learns the acoustic model based on the converted label string is provided.

The present invention has the following excellent effects.
According to the end-to-end Japanese speech recognition model learning apparatus according to the present invention, it is possible to learn a plurality of characters or a plurality of words with low appearance frequency as one class label.
Therefore, it is possible to learn acoustic features with a larger number of samples than a conventional method for a plurality of characters or a plurality of words having a low appearance frequency, and the speech recognition rate is improved.
In addition, such learning makes it possible to reduce duplication of learning network expressions caused by a large number of output labels such as Japanese, thereby improving the speech recognition rate.

1 is a block diagram schematically showing an end-to-end Japanese speech recognition model learning device according to a first embodiment of the present invention. It is a block diagram which shows typically the structure of the label preparation means of the end-to-end Japanese speech recognition model learning apparatus which concerns on 1st Embodiment of this invention. It is a schematic diagram of an acoustic model, (a) is a schematic diagram which outputs a label from an input audio | voice, (b) is a schematic diagram which also outputs a class label from the input audio | voice. (A) is an example of a character / label conversion table, (b) is a conceptual diagram of an acoustic model for converting characters into class labels, and (c) is a concept of a language model for restoring class labels in text to characters. FIG. It is a flowchart which shows the flow of the production process of the class label by the label production means concerning 1st Embodiment. It is a block diagram which shows typically the structure of the label preparation means of the end-to-end Japanese speech recognition model learning apparatus which concerns on 2nd Embodiment of this invention. (A) is a schematic diagram of an acoustic model that outputs a plurality of class labels from input speech, (b) is a conceptual diagram of an acoustic model that converts characters into class labels, and (c) is a class label in the text. It is a conceptual diagram of the language model which restore | restores to a character. It is a schematic diagram of a morpheme list and a reading list. (A) is a conceptual diagram which shows an example of the process by a speech recognition means, (b) is a schematic diagram of the converter which outputs the word which comprises the language model learning means which performs the speech recognition of (a). . It is a flowchart which shows the flow of the production process of the class label by the label production means which concerns on 2nd Embodiment. It is a flowchart which shows the flow of a process when a Chinese character is selected in the process of FIG. (A) is a schematic diagram of a flow of speech recognition processing using a conventional pronunciation dictionary, and (b) is a schematic diagram of a flow of conventional end-to-end speech recognition processing in English.

Hereinafter, a Japanese speech recognition model learning device according to an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
[Configuration of end-to-end Japanese speech recognition model learning device]
The end-to-end Japanese speech recognition model learning device 2 uses a character or a word (hereinafter simply referred to as “speech 1a”) which is learning data 1 including the text 1b and the speech 1a or the acoustic feature of the speech (hereinafter simply referred to as “speech”). An end-to-end acoustic model that outputs label 3 of “character” is learned as a speech recognition model.

  In the present embodiment, the learning data 1 for creating a Japanese acoustic model is described as a pair of a voice 1a and a text 1b. Speech 1a and text 1b represent a large amount of Japanese speech data and a large amount of text. For example, the program sound of a pre-learning broadcast program can be used as the sound 1a, and the text 1b can be a transcribed or equivalent content of the program sound. In FIG. 1, the converted text 1c, the label 3, and the label / conversion table 4 indicate data.

The end-to-end Japanese speech recognition model learning device 2 includes a label creating unit 20, a text creating unit 5, an acoustic model learning unit 6, a language model learning unit 7, an acoustic model storage unit 8, and a language model storage. And means 9. Here, the end-to-end Japanese speech recognition model learning device 2 includes speech recognition means 10.
The end-to-end Japanese speech recognition model learning device 2 represents, from the text 1b in the learning data 1, a plurality of characters whose appearance frequency included in the text 1b is lower than a predetermined appearance frequency standard and representing the plurality of characters. Based on the character / label conversion table 4 for creating a class label to be assigned to a class to be assigned and a single label for a single character having a higher appearance frequency than the reference, and assigning a plurality of characters to the class label, the learning data 1 A plurality of characters included in the text 1b are converted into class labels, and a post-conversion text 1c that is a text after the text 1b is converted is created. Then, the speech 1a, the converted text 1c, the class label, and a plurality of single labels in the learning data 1 are learned, and the speech 1a is converted into a class label and a single label label string by the learning, and the converted label string is converted into the converted label string. Based on this, the acoustic model is learned.

  The label creating unit 20 assigns a plurality of characters whose appearance frequency included in the text 1b is lower than a predetermined appearance frequency standard from the text 1b in the learning data 1 to a class representing the plurality of characters, And a single label relating to a single character having an appearance frequency higher than that of the reference. A single label (hereinafter referred to as a character label) and a class label are collectively referred to as a label 3. Label 3 is a label used for output of the acoustic model. The character label is a label that handles a single character, and the class label is a label that handles a plurality of characters as a group.

  The label creating means 20 creates and outputs a label 3 suitable for model learning from the text 1b and a character / label conversion table 4 which is a table regarding which character corresponds to which class label. Here, the label creating means 20 creates a character label from the text 1b in the learning data 1 and adds a class label later. The character labels include hiragana, katakana, kanji, alphabets, etc., as shown in the schematic diagram of FIG. The class label is indicated by an asterisk in the schematic diagram of FIG. 3B, but is not limited to this.

  In the present embodiment, the label creating means 20 includes a morpheme dividing means 21, a character list creating means 22, a label determining means 23, and a storage means 24, as shown in FIG.

  The morpheme dividing means 21 divides the text 1b of the learning data 1 into morphemes and gives a reading for each of them. As the morpheme dividing means 21, for example, general software for Japanese morpheme analysis (for example, MeCab or ChaSen) can be used. Hereinafter, the list obtained by dividing the text 1b of the learning data 1 into morpheme units is referred to as a morpheme unit list W. The morpheme list W is stored in the storage unit 24.

The character list creation means 22 counts the appearance frequency for each character in the text 1b of the learning data 1, and creates a list of characters whose appearance frequency is higher than a predetermined reference and a list of other characters. is there.
Here, a list composed of characters from the highest appearance frequency to N character types is referred to as a character list F. Further, a list including other low-frequency characters is referred to as a character list R. The character list F and the character list R are stored in the storage unit 24.

  The value of N can be set to a desired value, for example, can be made smaller than half of the total number of all character types in the text 1b of the learning data 1. In other words, the number of character types whose appearance frequency included in the text 1b in the learning data 1 is lower than a predetermined reference can be ½ or more of the total number of all character types included in the text 1b.

  The label determining unit 23 controls the entire label creating unit 20. The label determining unit 23 determines a character label for a high-frequency character, and determines a low-frequency character selected from the character list R as a class label. The label determining unit 23 stores the class label and the character label in the storage unit 24 as the label 3.

  Further, the label determining means 23 creates the label / conversion table 4 and outputs it to the text creating means 5. The label / conversion table 4 is a conversion table for assigning a plurality of characters to class labels. An example of the label / conversion table 4 is shown in FIG. This example corresponds to the schematic diagram of FIG. 3B, and the characters “璃, 鷲, Liu,...” Are respectively converted to the star “☆”.

  The storage unit 24 stores data created by the processing of the label creation unit 20 and is a general storage medium such as a hard disk. The storage means 24 stores morpheme list W, character list F, character list R, data of label 3, and the like.

  Based on the character / label conversion table 4, the text creation means 5 converts a plurality of characters included in the text 1b in the learning data 1 into class labels, and converts the converted text 1c, which is the text after converting the text 1b. create. The text creation means 5 inputs the text 1b, and uses the character / label conversion table 4 to rewrite the character group classified for each class label of the text 1b with the classified class label to convert the converted text 1c. create.

“Today's Owase is rainy” shown in the first line of FIG. 4B is an example of the text 1 b input to the text creation means 5.
4B is an example of the converted text 1c output by the text creating means 5 at this time.
The text input to the text creation means 5 may be a text different from the text 1b of the learning data 1.

  The acoustic model learning means 6 learns the speech 1a which is the learning data 1, the converted text 1c, the class label, and a plurality of single labels (character labels). A character string is converted into a label string, and an acoustic model is learned based on the converted label string. The acoustic model learning unit 6 learns a model that outputs which of the labels 3 is the voice using the label 3, the voice 1 a, and the converted text 1 c, and stores the model in the acoustic model storage unit 8. The acoustic model learning means 6 is applicable to all End-to-End acoustic models that specify a character sequence as described in Non-Patent Document 2.

  In this acoustic model, acoustic features (mel frequency cepstrum coefficients, filter bank output, etc.) extracted in advance from a large amount of speech data are classified into a deep neural network (Deep Neural Network) and a connectionist time series classification method (CTC) for each set label. : Connectionist Temporal Classification) etc. It should be noted that the likelihood calculation of the acoustic feature quantity by the acoustic model is a long short term memory (LSTM: Long Short Term) even if it is a recurrent neural network (RNN) if the output is a grapheme including Kanji. Memory).

  The language model learning means 7 learns the converted text 1c, and learns, as the speech recognition model, a language model that converts a label string of class labels and single labels (character labels) into a word string. The language model models an appearance probability of an output sequence (words and the like) learned in advance from a large amount of text. As this language model, for example, a general N-gram language model can be used.

  The language model learning means 7 learns a model that outputs a word string from the label 3 using the label 3 and the converted text 1 c and stores it in the language model storage means 9. The language model learning unit 7 uses the output of the acoustic model storage unit 8 as an input as in Non-Patent Document 2, and uses the acoustic model learning unit 6 to estimate and output a word string from the relationship between the preceding and following words. For characters that are not in the label 3, the characters are restored from the relationship between the character / label conversion table 4 and the preceding and following words.

The acoustic model storage unit 8 stores an acoustic model created by learning by the acoustic model learning unit 6 and is a general storage medium such as a hard disk.
The language model storage unit 9 stores a language model created by learning by the language model learning unit 7 and is a general storage medium such as a hard disk.

The speech recognition means 10 recognizes the input speech (evaluation speech) for each utterance section spoken by a person. The voice recognition means 10 outputs a word string as a recognition result to a display device or the like (not shown).
The voice recognition unit 10 converts the input voice into a feature amount (feature vector), and sequentially converts the feature amount into a label using an acoustic model stored in the acoustic model storage unit 8. Create a label column with. At this time, the voice recognition means 10 creates a label string such as “Today's tail ☆ city is rainy” shown in the first row of FIG.
Then, the speech recognition means 10 creates a word string by sequentially converting the label string into words using the language model stored in the language model storage means 9. At this time, the voice recognition means 10 creates a word string such as “Today's Owase is rainy” shown in the second row of FIG.

[Flow of class label creation processing]
The flow of class label creation processing by the label creation means 20 of the end-to-end Japanese speech recognition model learning device 2 according to the first embodiment will be described with reference to FIG.
First, the label creating means 20 of the end-to-end Japanese speech recognition model learning device 2 creates a morpheme unit list W obtained by dividing the text 1b of the learning data 1 into morphemes by the morpheme dividing means 21 (step S101).

Then, the label creating means 20 creates a character list F including the character list F of the top N appearance character types for each character in the text 1b of the learning data 1 and the character list R composed of other low-frequency characters by the character list creating means 22. (Step S102).
Then, the label creating means 20 selects the low-frequency character from the character list R by the label determining means 23 (step S103), adds the selected low-frequency character to the class label (step S104), and the character / label conversion table. 4 is updated (step S105).

  Then, the label determining unit 23 determines whether all the low frequency characters have been selected (step S106). If there is an unselected low-frequency character (step S106: No), the label determining means 23 returns to step S103. On the other hand, when all the low-frequency characters have been selected (step S106: Yes), the label creating means 20 creates a label 3 by integrating the class label and the character list F (step S107). Is output to the text creation means 5 and the processing is terminated.

  According to the present embodiment, by combining characters with low appearance frequency (low frequency characters) as one class label, the learning parameters are reduced and the number of learning samples per label is increased. It is easy to learn and has the effect of improving Japanese speech recognition accuracy.

(Second Embodiment)
Next, a Japanese speech recognition model learning apparatus according to a second embodiment of the present invention will be described with reference to FIG. Note that the end-to-end Japanese speech recognition model learning device according to the second embodiment differs from the first embodiment in that the label creating means 20B creates a plurality of class labels, but the other components are the first. Since it is the same as that of embodiment, drawing of the whole structure is abbreviate | omitted. Further, in the label creating means 20B shown in FIG. 6, the same components as those in the label creating means 20 shown in FIG.
The label creating means 20B creates a plurality of class labels by assigning a plurality of predetermined characters whose appearance frequency is lower than a predetermined criterion to any of a plurality of classes classified according to the predetermined criterion.

Here, as an example, the label creating means 20B reflects the phonological characteristics of a character and, as shown in FIG. 7A, is assigned to any of a plurality of classes divided for each character reading. We decided to create a class label. In FIG. 7A, for example, “<A>” assigned to a class whose reading is “A” is a character “A” indicating reading and two symbols “<” written on both sides thereof. "And">"represent class labels.
In this embodiment, when the text 1b input to the text creation means 5 is, for example, “Today's Owase is rainy” shown in the first line of FIG. 7B, the text creation means 5 As 1c, for example, a converted text 1c such as “Today's tail <wa> city is rainy” shown in the second line of FIG. 7B is output.
In the present embodiment, the voice recognition unit 10 creates a label string such as “Today's tail <wa> city is rainy” shown in the first line of FIG. 7C from the input voice. Using the language model, for example, a word string such as “Today's Owase is rainy” shown in the second row of FIG. 7C is created.
Hereinafter, each configuration of the label creating means 20B will be described with reference to FIG.

  As shown in FIG. 6, the label creating unit 20B includes a morpheme dividing unit 21, a character list creating unit 22, a label determining unit 23B, a storage unit 24, a morpheme list creating unit 25, and an edit distance calculating unit 26. , Reading delimiter estimating means 27 and reading list creating means 28 are provided.

  The morpheme list creating unit 25 creates a morpheme list J including the low-frequency characters in the character list R in the morpheme unit list W. The morpheme list J is stored in the storage unit 24. As the morpheme list creating means 25, for example, general software for Japanese morpheme analysis can be used.

In the present embodiment, the morpheme list creation unit 25 creates, based on the morpheme unit list W, a morpheme list J _s that is a list of morphemes including the kanji s for each kanji s of interest. The morpheme list J _s is stored in the storage unit 24 when processing for the Chinese character s.
For example, FIG. 8 shows an example of the morpheme list J _s created by the morpheme list creation unit 25 when the kanji s of interest is “raw”.

Further, here, the morpheme list creation means 25, character of all Chinese characters included in each morpheme j _s appearing in morpheme list J _s for each Chinese character s of interest (morphemes section j _s in the morpheme list J _s) A single kanji list, which is a list of all readings by itself, was also created. Specifically, when an example of the morpheme j _s is “raw”, the morpheme list creating unit 25 reads “sei”, “sho”, “ki”, “nam” as readings of “raw”, for example. Create the listed list.

Edit distance calculation means 26, for each morpheme j _s appearing in morpheme list J _s for each Chinese character s of interest, all combinations associated with the time you grant reading alone each Chinese character constituting the morpheme j _s The edit distance D _x is calculated by comparing with the reading j ^r _{s of the} entire morpheme j _s given by the morpheme dividing means 21.
Here, the edit distance D _x between the read combination of the Chinese character, a morpheme entire reading from one reading, inserting, deleting, by performing operations such substitutions, when editing the other readings, require Is the minimum number of operations to be taken. Edit distance calculation means 26 calculates the edit distance D _x by obtaining these deletion, insertion and substitution error characters.

Specifically, _assuming that an example of the morpheme j _s is “living organism” shown in FIG. 8, all combinations that are associated when a single reading is given to “raw” and “thing” respectively. , Obtained by combining the reading of each character.
Here, the reading of “raw” is, for example, “sei”, “sho”, “ki”, “nam”. In addition, the reading of “thing” is, for example, “butsu” or “thing”.
In this case, all combinations j ⁱ _{s, x} are “sei-mono”, “sei-butsu”, “sho-mono”, “sho-butsu”, “ki-mono”, “ki-butsu”, A total of eight combinations of “name-mono” and “name-butsu”.

The reading delimiter estimation means 27 obtains a kanji combination j ⁱ _{s, x} that minimizes the editing distance D _x and estimates a single reading delimiter j ^r _{s, s} of the kanji s of interest in the morpheme j _s . It is.
Overall read j ^r _s of "organism" shown in FIG. 8 is applied as "organism" in morphological analysis unit 21. However, the morpheme dividing means 21 gives a word-level reading, and in the symbol “living”, whether the reading of the symbol “raw” is “se” or “sei” There is no information. Therefore, the reading delimiter estimation means 27 has a high probability that the reading of the symbol “living” is “sei” in the symbol “living” based on, for example, the respective editing distances D _x for the total of the eight combinations described above. In the symbol “living”, a single reading break j ^r _{s, s} of “raw”, which is the kanji s of interest, is estimated.

The reading list creation means 28 refers to the single kanji list that is a list of all readings of one character alone, and reads the reading delimiter j ^r _{s, s} estimated for the k character s of interest in the morpheme j _s . Are all the readings j ⁱ _x of a single character of the target kanji s, and the morpheme j _s is classified for each reading j ⁱ _s of the target kanji s according to the determined reading. Stored in the read list L ^r _s .
This reading list L ^r _s is stored in the storage means 24 when processing for the Chinese character s.

In the example shown in FIG. 8, the reading of “raw” from the top to the third is “sei”, the reading of the fourth “raw” from the top is “sho”, and the fifth from the top The reading of “raw” is “yes”.
Therefore, in this case, the reading list creation means 28 stores “living things”, “students”, and “life” in the reading list L ^r _s corresponding to the reading “sei” of the kanji “raw”.
Further, the reading list creating means 28 stores “lifetime” in the reading list L ^r _s corresponding to the reading “sho” of the kanji “raw”.
Furthermore, the reading list creating means 28 stores “creature” in the reading list L ^r _s corresponding to the reading “i” of the kanji “raw”.

As with the label determining unit 23, the label determining unit 23B controls the entire label creating unit 20B, determines a high-frequency character as a character label, and determines a class label for a low-frequency character selected from the character list R. Determine as. The label determining unit 23B stores the class label and the character label in the storage unit 24 as the label 3.
When the low-frequency character is selected from the character list R, the label determining unit 23B assigns it to the corresponding class if it is not a Chinese character.
Label determining unit 23B, for each read j ⁱ list read and classified by _s L ^r _s of Kanji s of interest, the number of all morphemes j _s stored in the read list L ^r _s L ^r, a ^c _s, Count up and determine the initial letter of the reading list that has the maximum number of elements.

Specifically, in the example illustrated in FIG. 8, the number of morphemes L ^{r, c} _s stored in the reading list L ^r _s corresponding to the reading “sei” of the kanji “raw” is “3”.
In addition, the number of morphemes L ^{r, c} _s stored in the reading list L ^r _s corresponding to the reading “sho” of the kanji “raw” is “1”.
In addition, the number of morphemes L ^{r, c} _s stored in the reading list L ^r _s corresponding to the reading “i” of the Chinese character “raw” is “1”.
Therefore, in this case, since the reading list L ^r _s corresponding to the reading “sei” of the kanji “raw” has the maximum number of elements, the label determining means 23B uses the class “<” from the initial “se”. Determination> ”.

In the present embodiment, the storage means 24 stores a morpheme list J, a morpheme list J _s , and a reading list L ^r _s in addition to data such as a morpheme list W, a character list F, a character list R, and a label 3.

Here, the process in which the language model learning means 7 restores characters from the labels will be described with reference to FIGS. 9 (a) and 9 (b).
In FIG. 9A, as an example, the kanji characters “Kan”, “So”, “Meet” and “Send” are associated with “So”, which is the initial of the reading, and the class label “<So>”. The acoustic model which was assigned and learned is schematically shown. In addition, a language model that has been learned so that “concert”, which is an example of the original text 1b, can be restored from “come to the concert”, which is an example of the converted text 1c, is schematically shown. Yes.

  FIG. 9B shows a weighted finite state transducer (Weighted Finite State Transducer) for the word “concert” when learning by assigning the kanji “sou” to the class label “<so>” in the language model at this time. : WFST). The WFST is a converter that transitions information by writing a pair of input signals and output signals and their weights, and is also used in language model learning means in Non-Patent Documents 2 and 3. In FIG. 9B, description of transition probability is omitted from “input signal: output signal (transition probability)” on the arrow of WFST. “Eps” indicates a transition without input / output. “Space” indicates a blank transition.

  In the present embodiment, the language model learning means 7 includes a character string creating means (hereinafter referred to as a converter T), a word string creating means (hereinafter referred to as a converter L), and a sentence creating means (hereinafter referred to as a converter G). It is represented by the composition of each transducer. Here, decoding is performed by combining three converters: a CTC label-to-character converter T, a character-to-word converter L, and a word-to-sentence converter G.

The character string using the label 3 is generated by the converter T.
The converter L restores the Japanese word from the label string including the class label according to the character / label conversion table 4. As an example, FIG. 9B shows a word string output when the Chinese character “Kana” is assigned to the label “<so>”.
In the converter L, when converting the token string estimated by the converter T into a word, the converter L has a role of converting the class label assigned at the time of learning the acoustic model into a word including an original character.
In the converter G, a plausible recognition result is output from the word string candidates obtained in the converter L by statistical continuous information (n-gram) between words.

[Flow of class label creation processing]
Next, the flow of class label creation processing by the label creation means 20B of the end-to-end Japanese speech recognition model learning device 2 according to the second embodiment will be described with reference to FIG. Note that the processing in steps S101 and S102 shown in FIG. 10 is the same as the processing shown in FIG.

Subsequent to step S102, the label creating means 20B creates a morpheme list J including low-frequency characters in the character list R from the morpheme unit list W by the morpheme list creating means 25 (step S201).
Then, the label creating unit 20B selects a low-frequency character from the character list R by the label determining unit 23B (step S202).
Here, when the label determining means 23B selects a Chinese character as a low-frequency character, the label creating means 20B performs a process of estimating the reading of the Chinese character and adding it to the reading initial character list (step S203). Details will be described later.

If the hiragana or katakana character is selected as the low-frequency character, the label determining unit 23B adds it to the corresponding reading list (step S204).
Furthermore, when the number or alphabet that cannot be read is selected as the low-frequency character, the label determining unit 23B adds it to the unread list (step S205).
Alternatively, the label determining unit 23B adds a symbol having no reading as a low-frequency character to the symbol list (step S206).

Subsequent to any one of steps S203 to S206, the label creating means 20B updates the character / label conversion table 4 by the label determining means 23B (step S207). Then, the label determining unit 23B determines whether all the low-frequency characters have been selected (step S208). When there is an unselected low frequency character (step S208: No), the label determining unit 23B returns to step S202. On the other hand, when all the low-frequency characters are selected (step S208: Yes), the label 3 is created by integrating the class label set _Li and the character list F (step S209), and the character / label conversion table 4 is created as text. The data is output to the means 5, and the process is terminated.

  Next, processing when the label creating means 20B selects a Chinese character as a low-frequency character will be described with reference to FIG. 11 (refer to FIG. 10 as appropriate). Here, a part of the process shown in FIG. First, the label creating means 20B selects the kanji character s as the low-frequency character from the character list R by the label determining means 23B (initial value s = 1: step S202). Note that s (s = 1, 2,...) Is a code for identifying a Chinese character, but hereinafter simply referred to as a Chinese character s.

Then, the label creation means 20B uses the morpheme list creation means 25 to generate the morpheme list J _s including the Chinese character s (s-th Chinese character) from the morpheme list J including the low-frequency characters created in Step S201 (FIG. 10). Create (step S231).
Label creation means 20B is the label determining unit 23B, selects the morpheme j _s (j _s th morpheme) from the morpheme list J _s (step S232). _{Incidentally, j s (j s = 1,2} , ...) is a code for identifying the morphological, hereinafter, simply referred to as morphemes j _s.

Subsequently, the label creating means 20B estimates the reading j ^r _{s, s} of the Chinese character s in the morpheme j _s by, for example, the editing distance calculating means 26 and the reading delimiter estimating means 27, and the kanji character by the reading list creating means 28. The morpheme j _s is stored in the reading list L ^r _s prepared for each reading ^{r of} _s (step S233).

Then, the label determining unit 23B determines whether all morpheme sections have been selected (step S234). When there is an unselected morpheme section (step S234: No), the label determining unit 23B adds “1” to the value (j _s ) of the morpheme section (j _s = j _s +1: step S235), and step S232. Return to.

On the other hand, when all the morpheme sections are selected (step S234: Yes), the label creating unit 20B uses the reading list creating unit 28 to store the morphemes stored in the reading list L ^r _s classified for each reading r of the Chinese character s. The number L ^{r, c} _s of the reading ^r is counted, and the initial letter r _{t of the} reading r having the maximum number of elements in the reading list L ^r _s is obtained (step S236). Then, the label determining unit 23B adds the Chinese character s to the list L ^r _rt of the initial letter r _t (step S237).
However, if the reading cannot be estimated because the edit distance becomes larger than the specified value in step S236, it is added to the unread list as in step S205.

Then, the label creating means 20B updates the character / label conversion table 4 by the label determining means 23B (step S207). Then, the label determining unit 23B determines whether or not all the Chinese characters in the low-frequency character have been selected (step S208). If there is an unselected kanji section (step S208: No), the label determining unit 23B adds “1” to the value (s) of the kanji section (s = s + 1: step S238), and the process returns to step S202. On the other hand, when all the kanji sections in the low-frequency character are selected (step S238: Yes), the label determining unit 23B creates the label 3 by integrating the class label set _Li and the character list F (step S209), Outputs the label conversion table 4 to the text creation means 5 and ends the process. Incidentally, in step S209, if the Chinese character corresponding to in the class label set L _i is not present, the class labels may be omitted.

  According to the present embodiment, as in the first embodiment, there is an effect of improving Japanese speech recognition accuracy. In addition, since the character reading is divided into a plurality of class labels, the character reading is used as a hint when the character is restored from the class label, so that the word recognition accuracy is further improved.

  As mentioned above, although each embodiment of this invention was described, this invention is not limited to these, It can implement in the range which does not change the meaning. For example, the acoustic model learning unit 6 and the speech recognition unit 10 have been described as inputting speech and converting them into feature amounts internally, but acoustic feature amounts obtained by converting speech may be used as inputs.

  In each of the above embodiments, the end-to-end Japanese speech recognition model learning device has been described. However, end-to-end Japanese written in a general-purpose or special computer language so as to enable processing of the configuration of each device. It can also be regarded as a speech recognition model learning program.

Further, regarding the estimation of the initial reading, the morpheme list creating means 25 is a list of all the readings of each single kanji character included in each morpheme j _s that appears in the morpheme list J _s for each kanji s of interest. However, the present invention is not limited to this method. For example, it is possible to search one by one from the character reading list of the kakasi dictionary so as to match the front / back according to the position of the kanji and adopt the matched reading. Note that kakasi refers to a program and a dictionary created for the purpose of converting kanji-kana mixed sentences into hiragana and romaji sentences.

  Further, the end-to-end Japanese speech recognition model learning device 2 learns the end-to-end speech recognition model that directly outputs the character label 3 from the input speech 1a, but directly outputs the word label. You may do it. Although the number of Japanese words is larger than the number of Japanese character types, the number of parameters is also large. For example, it has been reported that a system that directly outputs words is possible even with a vocabulary number of about 100,000. ing.

  In the second embodiment, the reference for dividing into a plurality of class labels has been described as the reading of characters, but for example, it may be divided into a plurality of class labels on the basis of contexts such as character or word context and parts of speech.

In order to confirm the performance of the end-to-end Japanese speech recognition model learning apparatus according to the present invention, speech recognition experiments were conducted.
The word error rates when the speech recognition was performed using the acoustic model and the language model generated by the end-to-end Japanese speech recognition model learning device according to the first and second embodiments were respectively obtained. These are hereinafter referred to as Example 1 and Example 2. As a comparative example, the word error rate when speech recognition was performed without using a class label was obtained.

<Experimental conditions>
The Kaldi-based EESEN framework (https://github.com/srvk/eesen) was used. The Kaldi base is described in the following references.
(References)
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al., “The kaldi speech recognition toolkit, in Proc. ASRU, no. EPFL-CONF-192584. IEEE Signal Processing Society, 2011.

In the experiment, the Kaldi-based EESEN framework was modified as follows so that Japanese characters could be output.
The acoustic model is a CTC-based 4-layer BLSTM (Bi-directional Long Short Term Memory), and uses 712 hours of NHK (registered trademark) program audio and subtitle pairs as the audio 1a and the text 1b of the learning data 1. I learned.
The feature amounts are 120-dimensional feature parameters in total, and the breakdown is a 40-dimensional log mel filter bank feature amount and respective Δ and ΔΔ coefficients.
The number of LSTM memory cells is 320 in each direction.
The language model used was 3 gram WFST in ARPA format created from NHK (registered trademark) manuscript with a vocabulary of 200,000 words and subtitles.
NHK (registered trademark) information program “Hiruma Ehot” for 5 hours was used for the evaluation data.

Character types appearing in the above-mentioned 712 hours of data used as learning data 1 are kanji,
There are 3,476 species such as katakana. When these 3,476 character types were collected in descending order, the percentage of all characters in the learning data was investigated. As a result, almost all characters in the learning data are concentrated in the top 30% of high-frequency character types, and 99% of all characters appearing in the learning data can be covered with the top 42% 1,452 character types. I understood. In the experiment, as shown below, the top 1,500 character types were picked up as high-frequency character types.

As an output label of the acoustic model, an experiment was performed as a comparative example in which a 3,477 label obtained by adding a blank label to a label for all character types (3,476 types) appearing in the learning data 1 was output.
In Example 1, 1,500 high-frequency characters were extracted from the learning data 1 and used as 1,500 labels for outputting character labels. In addition, the other 1,976 characters were assigned to one class label. That is, 1,501 labels were used for acoustic model learning. In addition, a language model that restores the original character from the assigned class label was used.

  In the second embodiment, 1,500 high-frequency characters are extracted from the learning data 1 and used as 1,500 labels for outputting character labels. In addition, the other 1,976 characters were assigned to 73 class labels. Here, the 73 types are a class representing each reading of 46 character types including “O” and sound repellent in the Japanese syllabary diagram, a class representing each reading of 25 character types including muddy sound and semi-voiced sound, numbers and alphabets. This means an unreadable class such as etc. and a class of symbols. Actually, since class labels (3 class labels) to which none of 1,976 characters are assigned are excluded, only 1,570 labels are used for acoustic model learning. In addition, a language model that restores the original characters from the assigned 70 class labels was used. The experimental results are shown in Table 1.

  Compared to the case where the class is not used, the use of the class improves the speech recognition word error rate (Word Error Rate: WER). The class also improves WER by dividing all low frequency characters into multiple class labels rather than assigning them to one class label.

2 End-to-end Japanese speech recognition model learning device 5 Text creation means 6 Acoustic model learning means 7 Language model learning means 8 Acoustic model storage means 9 Language model storage means 10 Speech recognition means 20, 20B Label generation means 21 Morphological division means 22 Character list creation means 23, 23B Label determination means 24 Storage means 25 Morphological list creation means 26 Edit distance calculation means 27 Reading delimitation estimation means 28 Reading list creation means

Claims (5)

An end-to-end acoustic model that outputs a label of a character or a word from the speech or the acoustic feature amount of the speech, which is learning data including text and speech or the acoustic feature amount of the speech, is learned as a speech recognition model. End Japanese speech recognition model learning device,
From the text in the learning data, a plurality of characters or a plurality of words whose appearance frequency included in the text is lower than a predetermined appearance frequency standard are assigned to a class representing the plurality of characters or the plurality of words. Label creating means for creating a class label and a single label attached to a single character or single word having a higher appearance frequency than the reference;
Based on the conversion table that assigns the plurality of characters or the plurality of words to the class label, the plurality of characters or the plurality of words included in the text in the learning data are converted into the class label, and the text is converted. A text creation means for creating a post-conversion text that is a later text;
The speech that is the learning data or the acoustic feature amount of the speech, the converted text, the class label, and the plurality of single labels are learned, and the speech or acoustic feature amount is learned by the learning and the class label and An end-to-end Japanese speech recognition model learning apparatus comprising: an acoustic model learning unit that converts the label sequence of the single label and learns the acoustic model based on the converted label sequence.   The label creating means assigns a plurality of predetermined characters or a plurality of words whose appearance frequency is lower than a predetermined criterion to any of a plurality of classes classified according to a predetermined criterion. The end-to-end Japanese speech recognition model learning device according to claim 1, wherein   The label creating means assigns a plurality of predetermined characters or a plurality of words whose appearance frequency is lower than a predetermined reference to any of a plurality of classes divided for each character or word reading. The end-to-end Japanese speech recognition model learning device according to claim 2, wherein the label is created.   4. A language model learning unit that learns the converted text and learns, as the speech recognition model, a language model that converts the class label and the single-label label string into a word string by the learning. The end-to-end Japanese speech recognition model learning device according to any one of the above.   The program for functioning a computer as an end-to-end Japanese speech recognition model learning apparatus as described in any one of Claims 1-4.

Images