JP6546070B2 - Acoustic model learning device, speech recognition device, acoustic model learning method, speech recognition method, and program - Google Patents

Description

  The present invention relates to speech recognition technology, and more particularly to technology for learning an acoustic model used to recognize non-native speech.

  Speech recognition for non-native speech has acoustic characteristics that can not be found in native speech depending on the speaker's linguistic experience, native language, etc., such as reading errors and vowel insertion compared to speech recognition for native speech (See, for example, Non-Patent Document 1). Non-native speech recognition is compared to native speech recognition, as the uniqueness of these non-native speech adversely affects the discrimination performance of the discriminator (acoustic model) that performs acoustic score calculation to discriminate the phoneme of the input speech. Improvement in accuracy was a difficult task.

  Non-native GMM-HMM speech recognition is known as a technique for improving the recognition accuracy of non-native speech recognition (see, for example, Non-Patent Document 2). In the non-native GMM-HMM speech recognition, a label that manually evaluates the correctness of the pronunciation is added to the non-native speech data set, and the training data is divided based on the pronunciation evaluation value, Train multiple acoustic models by pronunciation level. This makes it possible to respectively specialize in differences in pronunciation derived from language experience, and improves speech recognition accuracy.

  In addition, there is non-native DNN-HMM speech recognition for speech recognition of non-native speech using a multilayer neural network acoustic model that achieves high recognition rates in all acoustic models of a speech recognition device (for example, non-patent document 3) reference).

Kawahara Tatsuya, Tsujimatsu Nobuaki, "Foreign Language Learning Support Using Speech Information Processing Technology", Transactions of the Institute of Electronics, Information and Communication Engineers D, vol. J96-D, no. 7, pp. 1549-1565, 2013 Takuya Ansai, Seitoshi Tsuji, Akinori Ito, "A Study on Acoustic Model Considering Pronunciation Level of Japanese English Learners", Proceedings of the Acoustical Society of Japan, 2011 Hiroshi Kibishi, Seiichi Nakagawa, "Recognition of Japanese English Speech by DNN-HMM", Proceedings of the Acoustical Society of Japan, 2013

  In the case of non-native speech recognition using the pronunciation evaluation value by the hand of the native teacher, it was necessary to manually set the pronunciation evaluation value for the speech data used when learning the acoustic model. The method of using the pronunciation evaluation value involves the problem that the pronunciation evaluation value is determined by the subjectivity and is not necessarily reliable, and not all utterances are evaluated based on the same standard, and the native teacher's hand There is a cost problem. Also, unlike GMM-HMM speech recognition, in DNN-HMM speech recognition using multilayer neural networks for acoustic models, there is no effective adaptation method like Maximum Likelihood Linear Regression (MLLR), and it is necessary to redo acoustic model learning is there. At this time, if the learning data is divided according to the pronunciation evaluation value, it is not possible to avoid the decrease in the recognition rate caused by the decrease of the learning data. Therefore, in DNN-HMM speech recognition, as in the case of GMM-HMM speech recognition, the recognition rate can be improved by using an approach that uses a pronunciation evaluation value.

  In view of the foregoing, it is an object of the present invention to provide a technique for learning an acoustic model that can recognize non-native speech with high accuracy, which is applicable even to DNN-HMM speech recognition. is there.

  In order to solve the above problems, the acoustic model learning device according to the first aspect of the present invention is extracted from non-native feature quantities representing non-nativeness of a speaker extracted from speech data for learning and speech data for learning A learning data storage unit for storing learning data in which a learning input feature amount combining an acoustic feature amount and transcription data representing utterance content of learning voice data is stored, and an acoustic model is generated using the learning data And an acoustic model learning unit to learn.

  A speech recognition apparatus according to a second aspect of the present invention extracts an acoustic model storage unit storing an acoustic model generated by an acoustic model learning apparatus, and extracts non-native feature quantities representing non-nativeness of a speaker from input speech data. Input voice input to the acoustic model input feature for recognition that combines non-native feature and acoustic feature, and non-native property extraction unit, acoustic feature quantity extraction unit for extracting acoustic feature from input voice data And a voice recognition unit for obtaining a voice recognition result of the data.

  The acoustic model learning technology of the present invention extracts non-native feature quantities that express objective non-nativeness without using human hands of a native teacher with linguistic expertise, and uses them as acoustic feature quantities. Learn acoustic models from combined training data. As a result, non-native speech can be recognized with high accuracy even in DNN-HMM speech recognition in which the speech recognition rate can not be improved conventionally using the pronunciation evaluation value.

FIG. 1 is a diagram illustrating a functional configuration of a learning data creation device. FIG. 2 is a diagram illustrating a processing procedure of a learning data creation method. FIG. 3 is a diagram showing a specific example of learning speech data. FIG. 4 is a diagram showing a specific example of learning data. FIG. 5 is a diagram illustrating a functional configuration of the acoustic model learning device. FIG. 6 is a diagram illustrating the processing procedure of the acoustic model learning method. FIG. 7 is a diagram illustrating the functional configuration of the speech recognition apparatus. FIG. 8 is a diagram illustrating the processing procedure of the speech recognition method.

  Hereinafter, embodiments of the present invention will be described in detail. In the drawings, components having the same functions are denoted by the same reference numerals and redundant description will be omitted.

  The embodiment of the present invention is a speech recognition system comprising the following three devices. The first apparatus is a learning data creation apparatus that adds non-native feature quantities extracted from learning speech data to acoustic feature quantities to generate learning data used for acoustic model learning. The second device is an acoustic model learning device that performs learning of an acoustic model using the learning data. The third device is a speech recognition device that adds non-native feature quantities extracted from input speech data to be recognized to acoustic feature quantities, and performs speech recognition using a learned acoustic model.

  These apparatuses are not necessarily configured by three units, and the apparatus configuration can be arbitrarily changed by changing the apparatus in which each component is arranged. For example, each unit of the learning data creation apparatus may be configured to be provided with an acoustic model learning apparatus, and one acoustic model learning apparatus may be implemented from learning data creation to acoustic model learning. Further, for example, each part of the learning data creation device and each part of the acoustic model learning device may be configured to be provided in the speech recognition device, and a single speech recognition device may be implemented from creation of learning data to sound recognition.

  Each device of the learning data creation device, the acoustic model learning device, and the speech recognition device according to the embodiment is, for example, a known device having a central processing unit (CPU), a main storage device (RAM: random access memory), etc. Or, it is a special device configured by loading a special program into a dedicated computer. Each device executes each process, for example, under the control of a central processing unit. Data input to each device and data obtained by each process are stored, for example, in the main storage device, and data stored in the main storage device is read out as needed and used for other processes . In addition, at least a part of each processing unit included in each device may be configured by hardware such as an integrated circuit. Each storage unit included in each device is, for example, a main storage device such as a random access memory (RAM), an auxiliary storage device configured by a semiconductor memory device such as a hard disk, an optical disk, or a flash memory, or a relational database And middleware such as a key value store.

  As shown in FIG. 1, the learning data generation device according to the embodiment includes a learning voice storage unit 10, a non-native extraction unit 11, an acoustic feature quantity extraction unit 12, a learning data generation unit 13, and a learning data storage unit 14. Including. The learning voice storage unit 10 and the learning data storage unit 14 do not necessarily have to be included in the learning data creation apparatus itself, and the learning voice storage unit 10 and the learning data storage unit 14 included in other external devices may be communication means such as a network. It can also be configured to be readable and writable via The acoustic model learning device performs the processing of each step shown in FIG. 2 to realize the learning data creation method of the embodiment.

  The learning speech storage unit 10 stores learning speech data used to learn an acoustic model. As shown in FIG. 3, the speech data for learning includes “identification number” for uniquely identifying each data, “voice data” representing a path to a voice file in which a non-native speaker's voice is recorded, voice data It is stored in association with "transcription data" in which the uttered content of T is transcribed.

  In step S11, the non-native extraction unit 11 extracts non-native feature quantities expressing non-nativeness of the speaker from the speech data of the speech data for learning. The extracted non-native feature amount is input to the learning data generation unit 13 in combination with the identification number of the speech data for learning.

  The non-native feature value is a continuous or discrete value or value that directly or indirectly reflects information specific to the non-native speaker, such as the language experience of the non-native speaker, correctness of the pronunciation, native language type, region of origin, etc. It is an amount expressed as a vector. The non-native extraction unit may be, for example, a discriminator, neural network, machine learning device, or the like, which is learned in advance to distinguish or evaluate the speech of the native speaker from the speech of the non-native speaker. May be used. At this time, intermediate processing results and outputs when speech is input to a machine learning apparatus such as a multi-layered neural network or SVM (Support Vector Machine) that performs discrimination, regression, and self coding may be used as non-native feature quantities. . As an intermediate processing result, for example, in a multi-layered neural network, output values of intermediate layers other than the final output layer may be used. For learning of a discriminator or the like, voice data of native speech or non-native speech, information on non-native speakers, information such as words of speech or phonemes may be used. The learning algorithm may be either supervised learning or unsupervised learning.

  As a specific example of the non-native feature, it is possible to output the score of the language determination result as the non-native feature by using a learned language discrimination model as a non-native extraction unit. As a score of a language discrimination result, it is a score value which shows each language likeness, for example. Another example of the score of the language determination result is an evaluation value of 0 to 1 in which the numerical value is closer to the first language and the numerical value is closer to 1 the closer to the second language. Also, the non-native extraction unit may have an acoustic model for native use (that is, an acoustic model that has learned native speech), and the score of the result of evaluating input speech data using this acoustic model may be a non-native feature. . As another example, the non-native extraction unit has a model for native speech recognition (that is, speech recognition that recognizes native speech), and the input speech data is recognized using this model. The recognition reliability may be a non-native feature.

  In step S12, the acoustic feature quantity extraction unit 12 extracts an acoustic feature quantity from the speech data of the speech data for learning. As the acoustic feature quantity, for example, mel frequency cepstrum coefficient or the one obtained by performing conversion such as normalization, or one obtained by combining a plurality of temporally preceding and succeeding feature quantities is used in acoustic model learning in speech recognition. Sound characteristic quantities and conversion results thereof. The extracted acoustic feature amount is input to the learning data generation unit 13 in combination with the identification number of the speech data for learning.

  In step S13, the learning data generation unit 13 has an identification number in which the non-native feature amount output by the non-native property extraction unit 11 and the acoustic feature amount output by the acoustic feature amount extraction unit 12 are combined with each feature amount. Combine to match, and generate input features for learning. The combination is a process of connecting one feature quantity to another feature quantity. In connection processing, the context of the two feature quantities is determined in advance. For example, when the acoustic feature "xxx" and the non-native feature "yyy" are extracted, "xxx yyy" obtained by connecting "xxx" and "yyy" in that order becomes the learning input feature. Thereafter, as shown in FIG. 4, the learning data generation unit 13 uniquely identifies each data as an “identification number”, the generated “input feature for learning”, and “transcription data” of speech data for learning. And generate learning data. The generated learning data is stored in the learning data storage unit 14.

  In the above-described embodiment, an example in which two feature quantities are combined to form learning data for performing acoustic model learning has been described. However, if only the condition that the acoustic feature quantity is included in the learning data is observed, two feature quantities are used. The process of obtaining learning data for performing acoustic model learning is not limited to this. For example, values obtained by inputting two feature quantities to a predetermined function may be added after (or before) the acoustic feature quantity. As a predetermined function, for example, normalization, or combining of a plurality of feature quantities that move in time may be performed, or may be input to another machine learning device learned in advance, and the intermediate processing result or the like The output may be used as the output of the function. In addition, after combining the acoustic feature amount and the non-native feature amount, one obtained by performing processing such as normalization and combining a plurality of frames may be used as learning data for performing acoustic model learning.

  In the above embodiment, an example in which an identification number is assigned to associate feature amounts, voice data, and transcription data is described, but instead of associating the identification number with each data, a non-native extraction unit and The same voice data is input to the acoustic feature quantity extraction unit, and generation data is generated without using identification number information by associating transcription data corresponding to the voice data with each feature quantity of the processing result. It is also possible to deform as it does.

  In the above-described embodiment, although the transcription data is directly acquired as corresponding to the teacher data used at the time of acoustic model learning, conversion is performed in advance to a different symbol format such as "symbol corresponding to phoneme" It is also good. For example, hiragana, katakana, phonemes, monophones, triphones, clustered triphones, state numbers, etc., may be converted into symbols representing readings or sounds, or numbers corresponding thereto. At that time, a symbol may be converted by a human or may be converted using another speech recognition decoder or an acoustic model. For example, forced alignment processing conventionally used in the DNN speech recognition field may be used or converted.

  The acoustic model learning device according to the embodiment includes, as shown in FIG. 5, a learning data storage unit 14, an acoustic model learning unit 15, and an acoustic model storage unit 16. The learning data storage unit 14 and the acoustic model storage unit 16 do not necessarily have to be included in the acoustic model learning device itself, and the learning data storage unit 14 and the acoustic model storage unit 16 included in other devices can be read and written via communication means such as a network. It is also possible to configure as possible. The acoustic model learning method of the embodiment is realized by the acoustic model learning device performing the processing of each step shown in FIG.

  The learning data storage unit 14 stores learning data generated by the learning data generation device. As described above, the learning data includes an identification number for uniquely identifying each data, a learning input feature amount obtained by combining a non-native feature amount extracted from speech data of learning utterance data, and an acoustic feature amount, and voice It is associated with transcription data in which the utterance content of data is transcribed.

  In step S15, the acoustic model learning unit 15 associates the learning input feature amount and the transcription data from the learning data stored in the learning data storage unit 14 and acquires the input data and uses the learning data for speech recognition. Learn acoustic model parameters. As a model represented by acoustic model parameters, for example, there is a model that converts a waveform into a symbol corresponding to a phoneme. As the “symbol corresponding to the phoneme”, for example, it is possible to create different acoustic models in advance, and use clustered triphones using the acoustic model, a state number representing the same, and the like.

  As shown in FIG. 7, the speech recognition apparatus according to the embodiment includes an acoustic model storage unit 16, a language model storage unit 20, a nonnative extraction unit 11, an acoustic feature quantity extraction unit 12, a feature quantity combination unit 21, and speech recognition Section 22 is included. The speech recognition method of the embodiment is realized by the speech recognition apparatus performing the process of each step shown in FIG.

  The acoustic model storage unit 16 stores an acoustic model including acoustic model parameters generated by the acoustic model learning device. The language model storage unit 20 stores a language model used for speech recognition.

  In step S11, the non-native extraction unit 11 extracts non-native feature quantities expressing non-nativeness of the speaker from the input speech data. The input speech data is speech data to be subjected to speech recognition, in which speeches by native speakers or non-native speakers are recorded. The non-native feature quantity extracted here is the same as the non-native feature quantity extracted by the learning data creation device. The extracted non-native feature amount is input to the feature amount combining unit 21.

  In step S12, the acoustic feature quantity extraction unit 12 extracts an acoustic feature quantity from the input voice data. The acoustic feature quantity extracted here is the same as the acoustic feature quantity extracted by the learning data creation device. The extracted acoustic feature amount is input to the feature amount combining unit 21.

  In step S21, the feature data combining unit 21 combines the feature data with the non-native feature data output by the non-native property extraction unit 11 and the acoustic feature data output by the acoustic feature data extraction unit 12. Combine in the same order as when the command is input to generate input feature quantities for recognition. The generated input feature quantity for recognition is input to the speech recognition unit 22.

  In step S22, using the acoustic model stored in the acoustic model storage unit 16, the speech recognition unit 22 outputs time-series data of "symbol corresponding to phoneme" from the input input feature amount for recognition. When the speech recognition unit has a language model that outputs speech recognition results (for example, text) from time series data of "symbol corresponding to phoneme", the output of the acoustic model is input to the language model, and the speech recognition result is output. Ru.

  The correctness label used for the learning of the non-native extraction unit may be directly used as the non-native feature amount instead of the output of the non-native extraction unit. The correct answer label may be, for example, information on the non-native speaker, such as language experience of the non-native speaker, correctness of pronunciation, native language type, region of origin, and the like. In speech recognition, non-native feature quantities estimated from input speech data may be used.

  The above embodiment describes an example in which the learning data creation device and the speech recognition device include the non-native extraction unit, but the non-native feature extraction device is an external device different from the learning data creation device and the speech recognition device. The non-native feature extraction device may extract the identification number and the voice data from the speech data for learning, and present the identification number and the non-native feature to the learning data creation device and the voice recognition device.

  In the above-described embodiment, only speech data by non-native speakers are used as learning speech data, but speech data by native speakers may be included in learning speech data and used. Specifically, native speech data is input to the non-native extraction unit, non-native feature quantities for native speech are calculated, and these are combined with acoustic feature quantities to generate learning data. Thereafter, the acoustic data may be learned using the learning data.

  As described above, the acoustic model learning technology of the present invention extracts non-native feature quantities that express highly objective non-nativeness without using the hands of a native teacher with linguistic expertise. The acoustic model is learned from the learning data in which B is combined with the acoustic feature. Further, according to the speech recognition technology of the present invention, non-native feature quantities are combined with acoustic feature quantities from speech data to be recognized to perform speech recognition using a learned acoustic model. With this configuration, non-native speech can be recognized with high accuracy even in DNN-HMM speech recognition in which the speech recognition rate can not be improved conventionally using the pronunciation evaluation value.

  The present invention is not limited to the above-described embodiment, and it is needless to say that changes can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment are not only executed chronologically according to the order described, but may be executed in parallel or individually depending on the processing capability of the apparatus executing the process or the necessity.

[Program, recording medium]
When various processing functions in each device described in the above embodiments are implemented by a computer, the processing content of the function that each device should have is described by a program. By executing this program on a computer, various processing functions in each of the above-described devices are realized on the computer.

  The program describing the processing content can be recorded in a computer readable recording medium. As the computer readable recording medium, any medium such as a magnetic recording device, an optical disc, a magneto-optical recording medium, a semiconductor memory, etc. may be used.

  Further, the distribution of this program is carried out, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD, a CD-ROM, etc. in which the program is recorded. Furthermore, this program may be stored in a storage device of a server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.

  For example, a computer that executes such a program first temporarily stores a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, at the time of execution of the process, the computer reads the program stored in its own recording medium and executes the process according to the read program. Further, as another execution form of this program, the computer may read the program directly from the portable recording medium and execute processing according to the program, and further, the program is transferred from the server computer to this computer Each time, processing according to the received program may be executed sequentially. In addition, a configuration in which the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes processing functions only by executing instructions and acquiring results from the server computer without transferring the program to the computer It may be Note that the program in the present embodiment includes information provided for processing by a computer that conforms to the program (such as data that is not a direct command to the computer but has a property that defines the processing of the computer).

  Further, in this embodiment, although the present apparatus is configured by executing a predetermined program on a computer, at least a part of the processing contents may be realized as hardware.

DESCRIPTION OF SYMBOLS 10 Voice memory unit 11 for learning Non-native property extraction unit 12 Acoustic feature quantity extraction unit 13 Learning data generation unit 14 Learning data storage unit 15 Acoustic model learning unit 16 Acoustic model storage unit 20 Language model storage unit 21 Feature quantity combination unit 22 Voice Recognition unit

Claims (6)

A learning input feature amount obtained by combining a non-native feature amount representing non-nativeness of a speaker extracted from learning voice data with an acoustic feature amount extracted from the learning voice data, and the utterance content of the learning voice data A learning data storage unit that stores learning data associated with transcription data representing
An acoustic model learning unit that learns an acoustic model using the learning data;
Only including,
The non-native feature value is the score of the intermediate processing result or language discrimination result of the language discrimination by the language discrimination model, the score of the intermediate processing result or evaluation result of the evaluation by the native acoustic model, or the confidence of the recognition result by the native speech recognition Is one of the degrees,
Acoustic model learning device. The acoustic model learning device according to claim 1, wherein
A non-native extraction unit for extracting the non-native feature amount from the learning voice data;
An acoustic feature amount extraction unit that extracts the acoustic feature amount from the learning voice data;
A learning data generation unit that combines the non-native feature amount and the acoustic feature amount to generate the learning input feature amount, associates the learning input feature amount with the transcription data, and generates the learning data When,
An acoustic model learning device further including An acoustic model storage unit storing an acoustic model generated by the acoustic model learning device according to claim 1 or 2 ;
A non-native soluble extract unit that extracts the non-native features from the input speech data,
An acoustic feature quantity extraction unit that extracts an acoustic feature quantity from the input voice data;
A speech recognition unit for inputting into the acoustic model an input feature quantity for recognition combining the non-native feature quantity and the acoustic feature quantity to obtain a speech recognition result of the input speech data;
Speech recognition device including. A learning data storage unit, a learning input feature amount obtained by combining a non-native feature amount representing non-nativeness of a speaker extracted from learning voice data and an acoustic feature amount extracted from the learning voice data; Learning data associated with transcription data representing utterance content of speech data for voice is stored,
Acoustic model learning unit, viewed contains an acoustic model learning step of learning an acoustic model by using the learning data,
The non-native feature value is the score of the intermediate processing result or language discrimination result of the language discrimination by the language discrimination model, the score of the intermediate processing result or evaluation result of the evaluation by the native acoustic model, or the confidence of the recognition result by the native speech recognition Is one of the degrees,
Acoustic model learning method. An acoustic model storage unit stores an acoustic model generated by the acoustic model learning method according to claim 4 ;
Non-native soluble extract portion, and a non-native of extracting the non-native features from the input speech data,
An acoustic feature quantity extraction step of extracting an acoustic feature quantity from the input voice data;
A speech recognition step in which a speech recognition unit inputs a recognition input feature quantity obtained by combining the non-native feature quantity and the acoustic feature quantity into the acoustic model to obtain a speech recognition result of the input speech data;
Speech recognition method including. A program for causing a computer to function as each part of the acoustic model learning device according to claim 1 or 2 or each part of a speech recognition device according to claim 3 .

Images