JP2020144213A - Program, device and method for pronunciation evaluation using inter-model distance - Google Patents

Abstract

To provide a program that can perform pronunciation evaluation with a small arithmetic processing load, for example, enables even real-time evaluation.SOLUTION: The present pronunciation evaluation program causes a computer to function as: inter-model distance calculation means which calculates for each of a plurality of pronunciation units (for example, phoneme) constituting a pronunciation of a predetermined language or predetermined pronunciation units among them, an inter-model distance between a model (for example, a hidden Markov model) structured for a pronunciation unit included in a pronunciation made by a distance calculation object and a model (hidden Markov model) structured for another pronunciation unit included in the pronunciation; evaluation score determination means which determines, based upon an inter-object model distance as the inter-model distance calculated targeting a pronunciation evaluation object as the distance calculation object, an evaluation score associated with discriminability of a pronunciation in pronunciation units made by the pronunciation evaluation object.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for evaluating the pronunciation of a predetermined language.

In recent years, with the development of Hidden Markov Models (HMM) and Automatic Speech Recognition (ASR) technology using Deep Neural Networks (DNN) algorithms, pronunciation evaluation and pronunciation scoring have been improved. The technology that implements automatically is drawing a lot of attention.

For example, Non-Patent Document 1 introduces some automatic pronunciation evaluation techniques using HMM. Specifically, as the first example, a technique for performing pronunciation evaluation using the posterior probability score of the utterance history acquired using a context-independent model set is disclosed along with the HMM forced alignment process. ing.

The technique uses 16000 utterances from 100 native speakers as training data for the acoustic model used to determine the pronunciation score. In addition, as the data necessary for pronunciation evaluation, the text of the utterance by the student who is the evaluation target is used.

As a second example, a technique for evaluating pronunciation using a segment duration score of pronunciation is disclosed. Here, the segment duration score is calculated using the log likelihood of the normalized duration derived based on the discrete distribution of the segment duration in the utterance. In addition, an HMM that outputs a discrete observation sequence that has been learned from pronunciation data native to the target language is used.

Further, as a third example, a technique for evaluating pronunciation using a timing score defined by a time interval at the center of each vowel in an intra-speech segment is disclosed.

Further, Non-Patent Document 2 discloses an automatic pronunciation evaluation technique using DNN. In this technology, pronunciation evaluation using GOP (Goodness Of Pronunciation) is performed, and this GOP is
(A) Posterior probabilities derived from the lattice by the forward backward algorithm,
It is calculated from (b) the average frame-level posterior probability derived from DNN, or (c) the likelihood ratio between the correct model and the competition model.

Of these, the GOP calculated from the DNN voice element (Senone, a unit finer than phonemes) node and the GOP calculated from the likelihood ratio between the correct answer model and the competition model are both the same as the human voice evaluation score. It shows a high correlation between them. By the way, the DNN learning dataset in this technology contains a 78-hour corpus recording the utterances of 284 native English speakers (Americans). Further, in this technology, accurate transcription data of utterances is utilized as necessary data.

H. Franco, L. Neumeyer, V. Digalakis, and O. Ronen, “Combination of machine scores for automatic grading of pronunciation quality”, Speech Communication, Volume 30, pp. 121-130, 2000 W. Hu, Y. Qian, and F. K. Soong, “A New DNN-based High Quality Pronunciation Evaluation for Computer-Aided Language Learning (CALL)”, In Proceedings of Interspeech 2013, pp. 1886-1890. 2013

However, in the prior art as described in Non-Patent Documents 1 and 2, there are serious problems regarding the processing load when constructing a model using learning data, the flexibility of the system, and the like. ..

For example, the technique described in Non-Patent Document 1 requires a huge amount of learning data, for example, thousands of utterance data by hundreds of native speakers, to construct an accurate HMM. ..

Furthermore, since this technique performs pronunciation evaluation based on forced alignment, accurate text information of utterances by pronunciation evaluation targets (for example, students) is required for the pronunciation evaluation. .. Therefore, the flexibility of the system is low, and for example, for a new pronunciation evaluation target (for example, a freshman), the model must be modified accordingly. Furthermore, due to such circumstances, the correlation coefficient between the evaluation score by this technology and the evaluation score by humans is only 0.579 at best.

Further, since the technique described in Non-Patent Document 2 is a method based on DNN, the setting is very complicated, and it is necessary to adjust many parameters in DNN. In order to obtain an accurate acoustic model, A huge amount of learning data is required. Further, this technique may also use forced cut-out type phoneme alignment, and in this case, as described above, accurate text information of the utterance by the pronunciation evaluation target is required in advance.

Further, in this technique, the load of arithmetic processing is considerably high, and for example, pronunciation evaluation in real time may become very difficult. In particular, in the case where the pronunciation evaluation is performed using the likelihood ratio between the correct answer model and the competition model, the arithmetic processing takes time, and therefore the pronunciation evaluation in real time is practically impossible. As a result, the evaluation score by this technology, the evaluation score by humans, and the correlation coefficient are only 0.50 at best.

Therefore, an object of the present invention is to provide a program, an apparatus, and a method capable of performing pronunciation evaluation with a smaller calculation processing load that also enables real-time evaluation.

According to the present invention, it is a pronunciation evaluation program for operating a computer mounted on a device for evaluating the pronunciation of a predetermined language performed by a pronunciation evaluation target by using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation An inter-model distance calculation means for calculating the inter-model distance with a model constructed for other included pronunciation units, and
Evaluation score determining means for determining the evaluation score regarding the distinctiveness of pronunciation of the pronunciation unit performed by the pronunciation evaluation target based on the target model distance, which is the distance between the models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation program is provided to make the computer function.

In the pronunciation evaluation program of the present invention, the evaluation score determining means is the distance between the target models and the distance between the reference models, which is the distance between the models calculated with the pronunciation reference target as the reference for the pronunciation evaluation as the distance calculation target. It is also preferable to determine the evaluation score based on the degree of difference.

Specifically, the evaluation score determining means is the difference between the distance between the target models and the distance between the reference models, and is based on the sum of the pronunciation units in the difference calculated for each pronunciation unit. It is also preferable to determine the evaluation score.

Further, the evaluation score determining means determines the evaluation score for the specific pronunciation unit based on the difference between the distance between the target models and the distance between the reference models, which is calculated for the specific pronunciation unit. It is also preferable.

Further, as an embodiment of the pronunciation evaluation program of the present invention, the pronunciation evaluation program uses the pronunciation of the pronunciation evaluation target for a model constructed using data related to the pronunciation performed by the pronunciation evaluation target equivalent. It is also preferable to further function the computer as a target model generation means for generating a model for the pronunciation unit included in the pronunciation performed by the pronunciation evaluation target by performing the regression adaptation processing.

Further, as another embodiment of the pronunciation evaluation program of the present invention, the pronunciation evaluation program refers to a model constructed by using data related to pronunciation performed by a pronunciation reference object that is a reference for pronunciation evaluation. By performing regression adaptation processing using the pronunciation of the evaluation target, the computer further functions as a target model generation means for generating a model for the pronunciation unit included in the pronunciation performed by the pronunciation evaluation target. It is also preferable.

Further, as a further embodiment of the pronunciation evaluation program of the present invention, the evaluation score determining means applies the value related to the distance between the target models to the trained evaluation score estimation model to determine the evaluation score. It is also preferable.

Further, in the pronunciation evaluation program of the present invention, it is also preferable that the model related to the pronunciation is a Hidden Markov Model. Further, it is also preferable that the pronunciation evaluation target is a learner of the predetermined language, and the evaluation standard target is a pronunciation provider who speaks the predetermined language as the mother tongue.

According to the present invention, it is also a pronunciation evaluation device that evaluates the pronunciation of a predetermined language performed by a pronunciation evaluation target by using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation An inter-model distance calculation means for calculating the inter-model distance with a model constructed for other included pronunciation units, and
Evaluation score determining means for determining the evaluation score regarding the distinctiveness of pronunciation of the pronunciation unit performed by the pronunciation evaluation target based on the distance between the target models, which is the distance between the models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation device having the above is provided.

According to the present invention, the pronunciation evaluation method is further performed by a computer mounted on a device for evaluating the pronunciation of a predetermined language performed by a pronunciation evaluation target using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation Steps to calculate the inter-model distance from the model constructed for the other included pronunciation units,
It has a step of determining an evaluation score regarding the distinctiveness of pronunciation of a pronunciation unit performed by the pronunciation evaluation target based on the target model distance, which is the distance between models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation method is provided.

According to the pronunciation evaluation program, apparatus and method of the present invention, pronunciation evaluation can be performed with less computational processing load that also enables real-time evaluation.

It is a schematic diagram which shows one Embodiment of the pronunciation evaluation system including the pronunciation evaluation apparatus by this invention. It is a schematic diagram for conceptually explaining the image of the phoneme distribution of native teachers and non-native students in the hidden Markov model (HMM) metric space, and the relationship between the distance between reference models and the distance between target models. It is a schematic diagram which shows schematic flow in one Embodiment of the pronunciation evaluation method by this invention. It is a radar chart for demonstrating the evaluation score determination process in one Example of the pronunciation evaluation method by this invention. It is a graph which shows the evaluation score determined in one Example of the pronunciation evaluation method by this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Pronunciation evaluation system / device]
FIG. 1 is a schematic diagram showing an embodiment of a pronunciation evaluation system including a pronunciation evaluation device according to the present invention.

The pronunciation evaluation system of the present embodiment shown in FIG. 1 is
(A) A pronunciation evaluation device 1 according to the present invention, which evaluates the pronunciation of a predetermined language performed by a pronunciation evaluation target using a model related to pronunciation such as a Hidden Markov Model (HMM).
(B) The pronunciation evaluation device 1 and the server 2 which are communicated and connected via the Internet or the like are included. Of these, the pronunciation evaluation device 1 of (a) can be, for example, a terminal that has downloaded the pronunciation evaluation program according to the present invention, that is, a personal computer (PC), a tablet computer, a smartphone, or the like.

The pronunciation evaluation device 1 inputs a pronunciation evaluation target, for example, a pronunciation group of a foreign language (predetermined language) being acquired by a student in a foreign language conversation class (for example, a learner of a second language) via a microphone 105, for example. , The student (pronunciation evaluation target) can be presented with an evaluation result for the pronunciation group, for example, a five-level evaluation score described later, via, for example, a display 107 or a speaker 106.

Specifically, the pronunciation evaluation device 1 is characterized by
(A) For each of the plurality of "pronunciation units" constituting the pronunciation of a predetermined language, or for the predetermined "pronunciation unit" among them, the "pronunciation unit" included in the pronunciation performed by the distance calculation target is constructed. The target model-to-model distance calculation unit 113 that calculates the "inter-model distance" between the model and the model constructed for other "pronunciation units" included in the pronunciation.
(B) Based on the "distance between target models", which is the distance between models calculated with the pronunciation evaluation target as the distance calculation target, the "evaluation score" regarding the distinctiveness of pronunciation of the pronunciation unit performed by the pronunciation evaluation target is determined. It has an evaluation score determination unit 114 and the like.

Here, the "pronunciation unit" of the above configuration (A) can be a "phoneme". For example, when the predetermined language is English, the number of "phonemes" in English is generally 44 (20 vowels, 24 consonants). For example, as will be described in detail later, each of the 44 "phonemes" For this model, calculating the "inter-model distance" from other (43) "phoneme" models and the sum of them is not a large processing burden.

Further, as a modification mode, for example, a "word" can be set as the "pronunciation unit" of the above configuration (A). However, in this case, since it is practically difficult to calculate the "inter-model distance" for all the "words" belonging to one predetermined language, the number of "words" constituting the pronunciation is limited to a predetermined number for pronunciation evaluation. Will be done.

More specifically, the model of the above configuration (A) can be an HMM for each "pronunciation unit", and the "inter-model distance" can be an HMM for one "pronunciation unit" and another. It can be the distance from the HMM for the "pronunciation unit". The specific definition of "inter-model distance" in HMM will be described in detail later in the description of the inter-model distance calculation unit 113.

Here, the inventors of the present application have found that the "distance between models" (total) calculated in this way can be used for pronunciation evaluation. For example, the sum of the "inter-model distances" between the HMM of one phoneme in a given language and the HMM of another (remaining) phoneme is greater for native speakers (those who speak the given language as their mother tongue). On the other hand, it tends to be smaller for non-native speakers.

This is because the more proficient in a given language, the more clearly the pronunciation of each phoneme is distinguished from the pronunciation of other (remaining) phonemes, that is, the more distinctive the pronunciation of each phoneme is. It depends. Therefore, as will be described in detail later with reference to FIG. 2, in general, the distribution of non-native phonemes in the HMM model metric space remains in a narrower range than that of native phonemes.

Based on such findings, the inventors of the present application devised the "evaluation score regarding the distinctiveness of pronunciation of pronunciation units" of the above configuration (B) by utilizing the "distance between target models".

Here, the processing load for calculating the "distance between models" is much smaller than that of the pronunciation evaluation processing by DNN, for example. Therefore, the above-mentioned "evaluation score" can be determined in real time (on the spot when the pronunciation of the pronunciation evaluation target is acquired) by constructing the necessary model in advance. That is, according to the pronunciation evaluation device 1, the pronunciation evaluation can be efficiently performed with a smaller calculation processing load that enables real-time evaluation.

Further, in the pronunciation evaluation device 1 of the present embodiment, it is not necessary to use the forced alignment method when performing the pronunciation evaluation. Therefore, for pronunciation evaluation, it is not necessary to convert the vocal data into text, it is not necessary for the evaluation standard target (for example, a native teacher) to provide the standard utterance sentence, and further, the evaluation standard target and the pronunciation evaluation target are the same. Alternatively, since it is not necessary to perform the corresponding pronunciation, the processing load of the pronunciation evaluation is further reduced.

As described above, in the present embodiment, it is possible to significantly reduce the amount of processing calculation and the amount of memory required in the pronunciation evaluation device 1, and in this case, for example, the pronunciation evaluation device 1 has a constant computing power. It can also be stored in a mobile terminal having a limit. Further, for example, a mode in which the "evaluation score" of the final pronunciation is output in substantially real time (with less delay time) can be realized.

By the way, the pronunciation evaluation target is naturally not limited to humans such as language learners, and may be, for example, an automatic dialogue system (providing a dialogue scenario) that enables dialogue in a predetermined language by learning. .. In addition, the evaluation criteria are not limited to humans such as native teachers. For example, it is possible to use an automatic dialogue system that enables native vocalization (sufficiently adopted as a criterion) by repeating updates. It is possible.

[Configuration of pronunciation evaluation device]
Similarly, according to the functional block diagram of FIG. 1, the pronunciation evaluation device 1 includes a communication interface unit 101, a native corpus 102, a user pronunciation storage unit 103, an evaluation score storage unit 104, a microphone (MC) 105, and a speaker. It has (SP) 106, a touch panel display (TP / DP) 107, and a processor memory.

Here, this processor memory stores one embodiment of the pronunciation evaluation program according to the present invention, has a computer function, and executes the pronunciation evaluation process by executing the pronunciation evaluation program. To do. For this reason, the pronunciation evaluation device 1 may be a dedicated pronunciation evaluation device, but may be a personal computer (PC), a notebook or tablet computer, a smartphone, or the like equipped with the pronunciation evaluation program according to the present invention. You can also do it.

Further, the processor memory includes a phoneme analysis unit 111, a target model generation unit 112 having an adaptation processing unit 112a, an inter-target model distance calculation unit 113, an evaluation score determination unit 114, an input / output control unit 131, and the like. It has a communication control unit 132. Further, an embodiment in which this processor memory has a reference model construction unit 121, a reference model distance calculation unit 122, and an evaluation score estimation model construction unit 123 is also possible.

It should be noted that these functional components can be regarded as the functions of the pronunciation evaluation program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the pronunciation evaluation device 1 in FIG. 1 with arrows is also understood as an embodiment of the pronunciation evaluation method according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the reference model construction unit 121 acquires pronunciation data of a plurality of predetermined languages for each teacher in the native teacher group that is the evaluation reference target from the native corpus 102, and these pronunciation data. Is used as training data to construct an HMM of a native teacher for each phoneme (pronunciation).

By the way, the learning data as described above may be acquired from an external server, for example, a native corpus owned by the server 2, for example, by the communication control unit 132 via the communication interface unit 101. In fact, the various native corpora that can be used as such learning data sources are provided by various associations and organizations, with a sufficient amount of learning data, for example, thousands of utterances by many natives. Is also feasible without much difficulty.

Further, as a modification, the pronunciation evaluation device 1 does not have the reference model construction unit 121, for example, the native teacher HMM constructed by the reference model construction unit 211 of the server 2 and the communication interface unit 101 by the communication control unit 132. You may get it through.

The reference model distance calculation unit 122 refers to each of a plurality of phonemes (pronunciation units) constituting the pronunciation of a predetermined language with respect to a specific phoneme included in the pronunciation performed by a native teacher (evaluation standard target). The "reference model distance", which is the sum of the model distances between the constructed HMM and the HMM constructed for other phonemes included in the pronunciation, is calculated.

Here, the distance between reference models for a specific phoneme can be regarded as an index indicating the relative position (relative to other phoneme groups) of this specific phoneme in the HMM model distance space of the native teacher, as described above. ..

In fact, the distance between reference models of native teachers (evaluation criteria) with higher phoneme pronunciation discrimination tends to be larger than that of non-native students (pronunciation evaluation). Therefore, the distance between reference models can be adopted as a criterion in pronunciation evaluation for non-native students. Moreover, since the sum of the distances is taken instead of a simple comparison between one specific phoneme and another phoneme, it is more suitable as a pronunciation evaluation standard.

Incidentally, the model distance D (λ _1, λ ₂₎ of HMMramuda ₁ and HMMramuda ₂ as the following equation _{(1) D (λ 1,} λ 2) = [log P (Q T (2) | λ 1) - log P (Q _T ⁽²⁾ | λ ₂ ))] / T ⁽²⁾
Can be adopted. The above equation (1) was proposed in Non-Patent Documents: BH Juang, LR Rabiner, “A probabilistic distance measure for hidden Markov models”, AT & T Technical Journal, Volume: 64, Issue: 2, pp.391-408, 1985. It was done.

Here, in the above equation (1), Q _T ⁽²⁾ is (the feature vector) of the observation sequence by HMM λ ₂ , T ⁽²⁾ is the total number of observations, and log P (Q | λ) is. It is a score for the observation sequence (feature vector) Q in HMMλ, that is, a score related to the probability that the observation sequence Q is output from HMMλ. This score is defined for evaluating the HMM, and can be used as an index showing the degree of agreement between the output of the HMM and the actually observed observation sequence. Therefore, as can be seen from the form of the above equation (1), the values of D (λ ₁ , λ ₂ ) and D (λ ₂ , λ ₁ ) are usually different from each other.

On the other hand, in the present embodiment, it is preferable to adopt an inter-model distance such that one distance value is determined if two HMMs for which the distance between them should be calculated are determined. That is, specifically, since it is preferable to adopt an inter-model distance that is symmetric with respect to the HMM λ _i of the phoneme _i and the HMM λ _{j of the} phoneme j in the native teacher, such an inter-model distance D _ij (= D _ji ) is used. The following equation (2) D _ij = [D (λ _i , λ _j ) + D (λ _j , λ _i )] / 2
The use, further, as the distance between the reference model phoneme i native teacher D _i ^(n), the following equation ^{_{(3) D i (n)}} = Σ j = 1, noti N D ij
To adopt. Here, N is the total number of phonemes in a predetermined language to be evaluated for pronunciation, and Σ _{j = 1, noti} ^N is the sum of 1 to N excluding i for j. The superscript (n) indicates that the value is related to the native teacher who is the target of the evaluation standard.

Incidentally, as a model distance D _ij of the above, it is the distance between the known HMM (a) the Euclidean distance between HMMramuda _j of HMMramuda _i phoneme j phoneme i, or (b) of HMMramuda _i phoneme j phoneme i It is also possible to adopt the Mahalanobis distance from HMMλ _j . However, by adopting the above equation (2) based on the above equation (1) as the inter-model distance _Dij , it is possible to calculate a more appropriate distance amount from the pronunciation data having a dynamic composition in the actual utterance. It will be possible.

Further, as a modification, the pronunciation evaluation device 1 does not have the reference model distance calculation unit 122 as described above, and is also based on the HMM of the native teacher constructed by the reference model construction unit 211 of the server 2, for example. The distance between the reference models calculated by the distance calculation unit 212 between the reference models of the server 2 may be acquired by the communication control unit 132 via the communication interface unit 101. Further, in accordance with such an aspect, a part of the component part (102-104, 111-114 in FIG. 1) of the pronunciation evaluation device 1 is omitted, and a part of the functions of the component part is mounted on the server 2, for example. You may. When the server 2 retains the functions of the distance calculation unit 113 between the target models and the evaluation score determination unit 114, the server 2 functions as the pronunciation evaluation device according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the user pronunciation storage unit 103 is pronunciation data of a predetermined language by a non-native student (pronunciation evaluation target) acquired, for example, via a microphone 105, and is input / output control unit 131. Save the pronunciation data converted to digital data in the specified format. Here, the user pronunciation storage unit 103 associates the pronunciation data with the identifier (ID) of the non-native student, classifies and stores it for each student, and uses the pronunciation data to store the HMM unique to the student for each student. It is also preferable to be able to generate.

The phoneme analysis unit 111 uses a known automatic speech recognition (ASR) technique to obtain phonemes and intervals of non-native students' pronunciations (data) acquired from the user pronunciation storage unit 103 or directly from the microphone 105. Divide into time (duration). Here, the text data of the pronunciation of non-native students is no longer necessary.

The target model generation unit 112
(A) Build an HMM using pronunciation data performed by speakers equivalent to non-native students.
(B) The adaptation processing unit 112a performs regression adaptation processing using the pronunciation of a specific non-native student who is the target of pronunciation evaluation on the HMM constructed in (a) above.
Generate HMMs for phonemes contained in pronunciations made by this particular non-native student.

Here, the utterance data in (a) above can be, for example, a number of data of about 4 to 6 obtained from the utterances of non-native students. That is, since the HMM constructed here is subsequently adjusted by the regression adaptation process, so much accuracy is not required.

Next, the regression adaptation process of (b) above can be carried out using, for example, a known MLLR (Maximum Likelihood Linear Regression). MLLR is an adaptation method that uses the weighted linear sum of regression matrices. Especially in the field of speech recognition, HMMs for general speakers (less speaker dependence) can be used for specific speakers. Alternatively, it is widely used to adapt to HMMs for specific environmental conditions. In fact, there are reports that the adaptation process with a very small number of utterance data produced an HMM comparable to the HMM constructed for a particular speaker or for a particular environmental condition.

In the present embodiment, the adaptation processing unit 112a uses the pronunciation data (corresponding frame) of a specific non-native student (pronunciation evaluation target) divided into phonemes by the phoneme analysis unit 111, and uses the above (a) by MLLR. ) Is adapted to this particular non-native student.

Here, the pronunciation data of the non-native student (pronunciation evaluation target) used in one adaptation process can be a very small number of data (for example, about 1 to 3). As a result, real-time regression adaptation processing and, by extension, pronunciation evaluation processing are also possible.

As a modification, the target model generation unit 112 is a pronunciation evaluation target in the adaptation processing unit 112a with respect to the native teacher HMM constructed by the reference model construction unit 121 (or constructed outside the device 1). It is also preferred to generate an HMM for the phonemes contained in the pronunciation performed by this particular non-native student by performing a regression adaptation process using the pronunciation of the particular non-native student. In this aspect, the above-mentioned “data relating to pronunciation performed by a speaker equivalent to a non-native student” in (a) becomes unnecessary, and it is also unnecessary to construct an HMM using the data.

Furthermore, the target model generation unit 112 constructs the HMM of this specific non-native student by using a large amount of pronunciation data for a specific non-native student (pronunciation evaluation target) without performing regression adaptation processing. It is also possible. However, by appropriately generating the HMM subjected to the regression adaptation processing as described above, it becomes easy to perform the pronunciation evaluation processing in real time, and the flexibility of the pronunciation evaluation processing, that is, various pronunciations. It is possible to increase the flexibility of responding immediately to the evaluation target (for example, the student who participated in the classroom for the first time).

Similarly, in the functional block diagram of FIG. 1, the target model-to-model distance calculation unit 113 has a specific non-specific phoneme (pronunciation unit) for each of a plurality of phonemes (pronunciation units) constituting the pronunciation of a predetermined language. The distance between the models of the HMM constructed for the phonemes included in the pronunciation performed by the native student (pronunciation evaluation target) and the HMM constructed for the other phonemes (pronunciation unit) included in the pronunciation. Calculate the total distance between target models.

Here, in the present embodiment, the distance between the target models D _i ⁽ⁿⁿ⁾ of the phoneme i of the specific non-native student is the same as the provision of the distance D _i ⁽ⁿ⁾ between the reference models in the above equations (2) and (3). Then, the following equation (4) D _i ⁽ⁿⁿ⁾ = Σ _{j = 1, noti} ^N [D (γ _i , γ _j ) + D (γ _j , γ _i )] / 2
To adopt. Here, γ _i is the HMM of the phoneme i in this particular non-native student, and γ _j is the HMM of the phoneme j. In addition, the superscript (nn) indicates that the value is related to the non-native student who is the target of pronunciation evaluation.

The distance D _i ⁽ⁿⁿ⁾ between the target models derived in this way is used together with the distance D _i ⁽ⁿ⁾ between the reference models to calculate the evaluation score, which will be described in detail later.

FIG. 2 is a schematic diagram for conceptually explaining the image of the phoneme distribution of native teachers and non-native students in the HMM metric space, and the relationship between the distance between reference models and the distance between target models.

Hereinafter, a virtual HMM metric space is assumed in which the HMMs of individual phonemes whose distances are defined are used as space points, and the distribution of individual phonemes (HMMs) in this HMM metric space is considered. .. Specifically, this HMM metric space is a coordinate space expressed by coordinates (α, β) having two feature parameters as coordinate components, and further, the predetermined language is Japanese and / a /, / i as phonemes. Take up the five vowels /, / u /, / e /, / o /.

FIG. 2A shows an image of the distribution of five vowels in the HMM metric space in a teacher who is native to Japanese. Here, the distance between the two vowels (between the black circles) corresponds to the distance between the models, and from these distributions, _Di ⁽ⁿ⁾ of the above equation (3 ⁾ is calculated for each vowel.

On the other hand, FIG. 2B shows an image of the distribution of five vowels in the HMM metric space in students who are non-native to Japanese. It can be seen that the distribution of non-native students remains in a narrower range than the distribution of native teachers shown in FIG. 2 (A). This reflects, as described above, that the lower the degree of proficiency with respect to a given language, the lower the distinctiveness of pronunciation of each phoneme.

Therefore, D _i ⁽ⁿⁿ⁾ shown in FIG. 2 (B) to the indicated non-native students above equation calculated for each vowel (4) is smaller than D _i ⁽ⁿ⁾ in FIG. 2 (A), i.e. In general, the following equation (5) D _i ⁽ⁿⁿ⁾ <D _i ⁽ⁿ⁾
Is understood to hold.

In this way, the distance between target models D _i ⁽ⁿⁿ⁾ can be used as a score for pronunciation evaluation by using the distance D _i ⁽ⁿ⁾ between reference models as an evaluation standard. Specifically, the smaller the distance D _i ⁽ⁿⁿ⁾ between the target models compared to the distance D _i ⁽ⁿ⁾ between the reference models, the lower the pronunciation evaluation.

Returning to the functional block diagram of FIG. 1, the evaluation score determination unit 114 is a phoneme performed by a non-native student based on the inter-model distance D _i ⁽ⁿⁿ⁾ calculated for the non-native student (pronunciation evaluation target). Determine the "evaluation score" for the distinctiveness of pronunciation. Here, the object model distance D _i ^(nn), the degree of difference between the reference model distance D _i ^(n), for example, the difference ^{_{(D i (nn) -D i}} (n)) or the ratio (D _i ⁽ it is preferable to determine the "rating score" based on the ^{_{nn) / D i (n)}} ).

Specifically, in this embodiment, the evaluation score determination unit 114, the difference between the distance between the object model is calculated for each phoneme _i D i ⁽ⁿⁿ⁾ a reference model distance ^{_{D i (n) (D i}} (nn) Based on the sum of the phonemes i in −D _i ⁽ⁿ⁾ ), that is, the following equation (6) S ＝ [Σ _{i = 1} ^N (D _i ⁽ⁿⁿ⁾ −D _i ⁽ⁿ⁾ ) ² ] ^0.5
Determines the score metric S. This score metric S is a metric for the entire pronunciation (all phonemes) that reflects the (non-) similarity between the pronunciation of each phoneme by a non-native student and the pronunciation of the phoneme by a native teacher.

As a metric different from the above S, the evaluation score determination unit 114 determines the score metric S _i for this specific phoneme i based on the distance between the models of the non-native student and the native teacher calculated for the specific phoneme _i. May be determined. This score metric S _i is specifically expressed by the following equation (7) S _i = [Σ _{j = 1, noti} ^N (D _ij ⁽ⁿⁿ⁾ −D _ij ⁽ⁿ⁾ ) ² ] ^0.5
D _ij ⁽ⁿⁿ⁾ = [D ⁽ⁿⁿ⁾ (λ _i , λ _j ) + D ⁽ⁿⁿ⁾ (λ _j , λ _i )] / 2
D _ij ⁽ⁿ⁾ = [D ⁽ⁿ⁾ (λ _i , λ _j ) + D ⁽ⁿ⁾ (λ _j , λ _i )] / 2
It is represented by. Here, D ⁽ⁿⁿ⁾ (λ _i , λ _j ) and D ⁽ⁿ⁾ (λ _i , λ _j ) are the above equations for non-native students (pronunciation evaluation targets) and native teachers (evaluation criteria targets), respectively. It is D (λ ₁ , λ ₂ ) of 1).

Both the score metrics S and S _i described above are smaller as the pronunciation of non-native students and the pronunciation of native teachers are similar. That is, the better the pronunciation of non-native students, the smaller the value ⁽ contrary to the tendency of the distance D _i ⁽ⁿⁿ⁾ between target models ⁾ . As a modification, it is also possible to determine the score metric such as S and S _i described above without using the inter-model distance for the native teacher (evaluation standard target). For example, (Σ _{i = 1} ^N D _i ⁽ⁿⁿ⁾ ) and (Σ _{j = 1, noti} ^N D _ij ⁽ⁿⁿ⁾ ) can be score metrics. Also in this case, of course, the score metric is determined based on the distance between the target models for the non-native student (pronunciation evaluation target).

The evaluation score determination unit 114 then determines the final "evaluation score" for the non-native student's pronunciation of a predetermined language using the score metric S or S _i determined as described above.

Specifically, as one embodiment, the evaluation score estimation model construction unit 123
(A) Score metric S (or S _i ) and
(B) An evaluation score estimation model is constructed from a plurality of teacher data that are a set of evaluation scores by human evaluators as correct answer values, and then the evaluation score determination unit 114 uses the constructed evaluation score estimation model. It is used to determine the evaluation score. That is, the score metric S (or S _i ) of the non-native student (value related to the distance between models) to be evaluated for pronunciation is applied to this learned evaluation score estimation model to determine the evaluation score.

Here, various formats and standards can be adopted for the evaluation score, for example, "Native level" (Agree level), "Mildly Agree level", and "Undecided". It can be a value indicating five levels of level), "quasi-non-native level" (Mildly Disagree level), and "non-native level" (Disagree).

Of course, this evaluation score indicates the overall level of pronunciation of non-native students (pronunciation evaluation target) when score metric S is used, while non-native when score metric S _i is used. It indicates the pronunciation level of the phoneme i of the student (pronunciation evaluation target). When deriving both evaluation scores, different evaluation score estimation models corresponding to each are constructed.

Further, the evaluation score estimation model constructed here may be a regression model or another machine learning model. Incidentally, as a modification mode, it is also possible that the server 2 constructs the evaluation score estimation model in the evaluation score estimation model construction unit 213 and provides the evaluation score estimation model to the pronunciation evaluation device 1. In this case, the evaluation score estimation model construction unit 123 becomes unnecessary.

Further, the evaluation score determination unit 114 displays the determined evaluation score (for example, the above-mentioned five-step score) via the input / output control unit 131, for example, by displaying it on the display 107 or by the voice output from the speaker 106. May be presented to. Further, it is also preferable to associate the evaluation score with the identifier (ID) of the corresponding non-native student and the evaluation period information and store it in the evaluation score storage unit 104. In this case, for example, it is possible to grasp and present the past evaluation status of this non-native student and the transition of the evaluation score (language learning progress status).

Further, it is also preferable that the evaluation score determination unit 114 transmits the determined evaluation score to an external information processing device, for example, a student data management server in a language learning classroom, via the communication control unit 132 and the communication interface unit 101.

[Pronunciation evaluation method]
FIG. 3 is a schematic diagram schematically showing a flow in one embodiment of the pronunciation evaluation method according to the present invention.

As shown in FIG. 3, the pronunciation evaluation method of the present embodiment is composed of two modes, an offline mode and an online mode. Of these, in offline mode
(S1) The phoneme analysis unit 111 acquires pronunciation data of a large number of native teachers from the native corpus, decomposes each pronunciation data into phonemes, and then obtains learning data for constructing a phoneme HMM (HMM related to each phoneme). Generate.

(S2) The reference model construction unit 121 constructs a phoneme HMM related to the pronunciation evaluation standard for the native teacher by using the learning data generated in step S1.
(S3, S4) The reference model distance calculation unit 122 calculates the phoneme HMM distance using the phoneme HMM constructed in step S2, and further calculates the reference model distance using the calculated phoneme HMM distance. To do.

(S5) Further, in the offline mode of the present embodiment, the target model generation unit 112 constructs a phoneme HMM for non-specific students using the pronunciation data of the non-native student prepared in advance, and generates the target model in the online mode. Prepare for processing (MLLR adaptation processing).

On the other hand, in online mode
(S6) The phoneme analysis unit 111 acquires pronunciation data of a non-native student who is a pronunciation evaluation target via, for example, a microphone 105, decomposes the pronunciation data into phonemes, and then performs MLLR processing for the subsequent operation. Generate data.
(S7) The target model generation unit 112 performs MLLR processing on the phoneme HMM for non-specific students constructed in the offline mode using the data generated in step S6, and performs MLLR processing on the specific phoneme HMM for pronunciation evaluation. Generate phoneme HMMs for non-native students.

(S8, S9) The target model inter-model distance calculation unit 113 calculates the inter-phoneme HMM distance using the phoneme HMM generated in step S7, and further calculates the inter-target model inter-model distance using the calculated inter-phoneme HMM distance. To do.

(S10) Finally, the evaluation score determination unit 114 refers to a specific non-native student (pronunciation evaluation target) from the target model distance calculated in step S9 based on the reference model distance calculated in advance in the offline mode. (For example, the above-mentioned five-step score) is determined.

As described above, in the pronunciation evaluation method of the present embodiment, the distance between the reference models and the phoneme HMM for non-specific students are prepared in advance in the offline mode, and further, in the online mode, MLLR processing is used to use a small number of pronunciation data. And under a small processing load, the phoneme HMM of a specific non-native student (pronunciation evaluation target) can be quickly generated and the evaluation score can be determined. As a result, for example, it is possible to implement this pronunciation evaluation method on a mobile terminal having a certain limit in calculation ability and present the evaluation score of a non-native student (pronunciation evaluation target) to the non-native student in almost real time. It becomes.

[Example]
FIG. 4 is a radar chart for explaining the evaluation score determination process in one embodiment of the pronunciation evaluation method according to the present invention. Here, FIGS. 4 (A) and 4 (B) show the phoneme / b / for the students A and B, which are the pronunciation evaluation targets, and the other plosives j (/ p /, / t /, / d /, respectively. It is a radar chart showing the inter-model distance D _{/ b /, j} ⁽ⁿⁿ⁾ with / k /, / g /).

From the distribution of the inter-model distance D _{/ b /, j} ⁽ⁿⁿ⁾ of student A shown in FIG. 4 (A), the score metric S _{/ b /} for the phoneme / b / of student A is calculated to be 13.5 ( Using the above equation (7), Σ _j is calculated as the sum of the above five plosive sounds j). On the other hand, from the distribution of the inter-model distance D _{/ b /, j} ⁽ⁿⁿ⁾ of student B shown in FIG. 4 (B), the score metric S _{/ b /} for the phoneme / b / of student B was calculated to be 16.5. It is larger than the score metric S _{/ b /} (= 13.5) of Student A. That is, the pronunciation of the phoneme / b / of the student B is more dissimilar to that of the standard native teacher / b / as compared with that of the student A.

This is because the inter-model distance D _{/ b /, j} ⁽ⁿⁿ⁾ of student A is larger overall than that of student B, that is, in the graph line of the radar chart of FIG. 4 (A). This is a result reflecting that the surrounding area is larger than the area shown in FIG. 4 (B). From this, it can be seen that the pronunciation of the phoneme / b / by Student A is more proficient (closer to the native) than that of Student B.

FIG. 5 is a graph showing the evaluation score determined in one embodiment of the pronunciation evaluation method according to the present invention. Here, in FIG. 5 (A), score metrics S _{/ a /} , S _{/ i /} and S _{/ z /} for each pronunciation of vowel / a /, vowel / i / and consonant / z / by student A, In addition, five-grade evaluation scores for vowel / a /, vowel / i / and consonant / z / are shown, while FIG. 5 (B) shows vowel / a /, vowel / i / and vowel / i / by student B. Evaluation scores for each consonant / z / pronunciation S _{/ a /} , S _{/ i /} and S _{/ z /} , and a five-point evaluation score for the vowel / a /, vowel / i / and consonant / z / are shown. ing.

Incidentally, in this embodiment, the evaluation score estimation model for determining the five-stage evaluation score is not a regression model but a model using accumulated score metric statistics. Specifically, first, the phonemes / a /, / i / and / z / HMMs for native teachers, and the phonemes / a /, / i / and / z / HMMs for some non-native students. For each of the phonemes / a /, / i / and / z /, the maximum value S _MAX and the minimum value S _MIN of the score metric (S _{/ a /} , S _{/ i /} , S _{/ z /} ) are generated. And have been decided.

Then set this minimum S _MIN to "Agree level", set the maximum S _MAX to "Non-native level" (Disagree), and finally these minimum S _MIN and maximum. between the S _MAX (the section length (S _MAX -S _MIN) / 4 to become) four three section boundary position generated when the equally divided sequentially into sections, remaining "quasi native level" (Mildly It is set to Agree level), "Undecided level" and "Mildly Disagree level".

First, according to FIG. 5 (A), Student A has S _{/ a /} = 5.447 and S _{/ i /} = 6.045 for the vowels / a / and / i /, which is generally "quasi-native level" ( It has a Mildly Agree level) rating score. On the other hand, for consonants / z /, S _{/ z /} = 8.500, which is slightly lower than the vowels / a / and / i /, "undecided level" and "quasi-native level". It is an evaluation score between "(Mildly Agree level).

By the way, the phoneme accuracy acc of Student A calculated in the series of phoneme analysis processes related to the above evaluation result was 70.3%. Here, the phoneme pronunciation accuracy acc is calculated by the following equation (8) acc = (N−S−D−I) / N.
Is calculated using. In the above equation (8), N is the total number of phoneme (column) cases acquired, S is the number of phoneme substitutions (substitutions) as pronunciation errors, and D is necessary as a pronunciation error. It is the number of occurrences of absence / deletion of phonemes, and I is the number of occurrences of insertion of unnecessary phonemes as a pronunciation error.

Next, according to FIG. 5 (B), student B has S _{/ a /} = 7.503 and S _{/ i /} = 7.786 for the vowels / a / and / i /, which is an "average level" (Undecided). He has earned an evaluation score between level) and the "Mildly Agree level". On the other hand, for the consonant / z /, S _{/ z /} = 13.500, which is lower than the vowels / a / and / i /, and is generally evaluated as "Mildly Disagree level". It is a score.

From these results, the pronunciation of the phonemes / a /, / i / and / z / of student B is less proficient than that of student A (phoneme pronunciation based on the native pronunciation). The degree of error occurrence is higher). By the way, the phoneme pronunciation accuracy acc of student B is 65.2%, which is lower than the phoneme pronunciation accuracy acc (70.3%) of student A, and the evaluation scores of both are shown in FIGS. 5 (A) and 5 (B). It supports the result of.

That is, in Student B, in which the degree of discrimination between phoneme pronunciations is lower (the distance between models is smaller) and phoneme pronunciations are more likely to be confused than in Student A, more errors in phoneme pronunciations occur, and further. From such a result, it is understood that the above evaluation score using the distance between the models is a very good pronunciation evaluation index.

As described in detail above, according to the present invention, the evaluation score is derived for the pronunciation evaluation target by using the distance between the target models. Here, the processing load for calculating the distance between the target models is much smaller than that of the pronunciation evaluation processing by DNN, for example. Therefore, this evaluation score can be determined in real time (on the spot when the pronunciation of the pronunciation evaluation target is acquired) by constructing the required model in advance. That is, according to the present invention, pronunciation evaluation can be efficiently performed with a smaller calculation processing load that also enables real-time evaluation.

Further, in the pronunciation evaluation device 1 of the present embodiment, it is not necessary to use the forced alignment method when performing the pronunciation evaluation. Therefore, it is not necessary to convert the utterance data into a text for the pronunciation evaluation, and it is not necessary to provide the standard utterance sentence by the evaluation standard target, and as a result, the processing burden of the pronunciation evaluation is further reduced.

In addition, the present invention makes it possible to evaluate appropriate language proficiency of individual learners at a lower burden, especially when providing language education services in a language school or during a public education period. .. Further, in general, in a language, a limited number of pronunciation units (for example, dozens of phonemes such as vowels and consonants) are clearly defined, and in the present invention, it is only necessary to calculate the inter-model distance for these. Since the score can be determined, it can be easily applied to pronunciation evaluation in various languages.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the claims and their equivalents.

1 Pronunciation evaluation device 101 Communication interface section 102 Native corpus 103 User pronunciation storage section 104 Evaluation score storage section 105 Microphone (MC)
106 Speaker (SP)
107 Touch panel display (TP / DP)
111 Phoneme analysis unit 112 Target model generation unit 112a Adaptation processing unit 113 Target model distance calculation unit 114 Evaluation score determination unit 121, 211 Reference model construction unit 122, 212 Reference model distance calculation unit 123, 213 Evaluation score estimation model construction Unit 131 Input / output control unit 132 Communication control unit 2 Server

Claims (11)

It is a pronunciation evaluation program that operates a computer mounted on a device that evaluates the pronunciation of a predetermined language performed by a pronunciation evaluation target using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation An inter-model distance calculation means for calculating the inter-model distance with a model constructed for other included pronunciation units, and
Evaluation score determining means for determining the evaluation score regarding the distinctiveness of pronunciation of the pronunciation unit performed by the pronunciation evaluation target based on the target model distance, which is the distance between the models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation program characterized by the functioning of a computer. The evaluation score determining means is based on the degree of difference between the distance between the target models and the distance between the reference models, which is the distance between the models calculated by using the pronunciation reference target as the reference for pronunciation evaluation as the distance calculation target. The pronunciation evaluation program according to claim 1, wherein the evaluation score is determined. The evaluation score determining means determines the evaluation score based on the difference between the distance between the target models and the distance between the reference models and the sum of the pronunciation units in the difference calculated for each pronunciation unit. The pronunciation evaluation program according to claim 2, wherein the pronunciation evaluation program is performed. The evaluation score determining means is the difference between the distance between the target models and the distance between the reference models, and determines the evaluation score for the specific pronunciation unit based on the difference calculated for the specific pronunciation unit. 2. The pronunciation evaluation program according to claim 2. By performing regression adaptation processing using the pronunciation of the pronunciation evaluation target on the model constructed using the data related to the pronunciation performed by the pronunciation evaluation target, the pronunciation performed by the pronunciation evaluation target can be obtained. The pronunciation evaluation program according to any one of claims 1 to 4, wherein a computer is further functioned as a target model generation means for generating a model for the included pronunciation unit. By performing regression adaptation processing using the pronunciation of the pronunciation evaluation target on the model constructed using the data related to the pronunciation performed by the pronunciation reference target that is the standard of the pronunciation evaluation, the pronunciation evaluation target The pronunciation evaluation program according to any one of claims 1 to 4, wherein a computer is further functioned as a target model generation means for generating a model for the pronunciation unit included in the performed pronunciation. .. The evaluation score determining means applies the value related to the distance between the target models to the trained evaluation score estimation model to determine the evaluation score, according to any one of claims 1 to 6. Described pronunciation evaluation program. The pronunciation evaluation program according to any one of claims 1 to 7, wherein the model related to the pronunciation is a Hidden Markov Model. The pronunciation evaluation target is a learner of the predetermined language, and the evaluation standard target is any one of claims 1 to 8, characterized in that the pronunciation evaluation target is a pronunciation provider who speaks the predetermined language as the mother tongue. Described pronunciation evaluation program. It is a pronunciation evaluation device that evaluates the pronunciation of a predetermined language performed by a pronunciation evaluation target using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation An inter-model distance calculation means for calculating the inter-model distance with a model constructed for other included pronunciation units, and
Evaluation score determining means for determining the evaluation score regarding the distinctiveness of pronunciation of the pronunciation unit performed by the pronunciation evaluation target based on the distance between the target models, which is the distance between the models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation device characterized by having and. It is a pronunciation evaluation method performed by a computer mounted on a device that evaluates the pronunciation of a predetermined language performed by a pronunciation evaluation target using a model related to pronunciation.
For each of the plurality of pronunciation units constituting the pronunciation of the predetermined language, or for the predetermined pronunciation unit among them, the model constructed for the pronunciation unit included in the pronunciation performed by the distance calculation target and the pronunciation Steps to calculate the inter-model distance from the model constructed for the other included pronunciation units,
It has a step of determining an evaluation score regarding the distinctiveness of pronunciation of a pronunciation unit performed by the pronunciation evaluation target based on the target model distance, which is the distance between models calculated with the pronunciation evaluation target as the distance calculation target. A pronunciation evaluation method characterized by this.

Images