JP2020057129A - Program, device and method for pronunciation assessment using language identification model - Google Patents

Abstract

To provide a program capable of assessing pronunciation of a prescribed language by an assessment object independently of a speech recognition technique imposing a heavy processing load because of high accuracy.SOLUTION: The program for assessing pronunciation of a prescribed language by an assessment object causes a pronunciation assessment device 1 to function as: reference score distribution acquisition means which acquires score distribution information of an assessment reference object determined by acquiring a plurality of scores for pronunciation of the prescribed language by the assessment reference object, each of which is acquired by using a language identification model which outputs a degree of certainty that input pronunciation is pronunciation in the prescribed language; object score distribution determination means which uses the same language identification model to acquire a plurality of scores for pronunciation of the prescribed language by the assessment object and determines score distribution information of the assessment object; and an assessment score determination means which determines an assessment score for pronunciation of the prescribed language by the assessment object on the basis of a value relating to a distribution parameter in a distribution of differences between the score distribution information of the assessment object and the score distribution information of the assessment reference object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for evaluating pronunciation in a predetermined language.

  In recent years, with the development of Automatic Speech Recognition (ASR) using a deep neural network (DNN, Deep Neural Networks) algorithm, a technique for automatically executing speech evaluation and pronunciation scoring has attracted attention.

  For example, Non-Patent Document 1 discloses a method of automatic pronunciation scoring based on DNN and a Gaussian Mixture Model (GMM). In this method, a utterance sentence of a student having the same phrase structure and a reference generation sentence of a teacher are adopted, and scoring is performed using a phoneme array and a likelihood ratio. The pronunciation score is given at the phoneme level, and indicates to what extent the student's utterance can imitate the teacher's utterance.

  Further, in this method, a bottleneck feature in the DNN is extracted, and a GMM-HMM (Hidden Markov Model) triphone acoustic model is constructed using the feature and used for phoneme alignment processing. According to Non-Patent Document 1, a score obtained by such a method is compared with a score by a human evaluator, and the correlation coefficient between the two has reached 0.717. Furthermore, the method is compared with the baseline method, and it is said that better results were obtained.

  For example, Non-Patent Document 2 discloses a pronunciation scoring method using ASR. In this method, a large amount (for example, 800 hours) of non-native (speakers whose native language is not the target language) corpus is used, and ASR by GMM, ASR by DNN, and tandem-type bottleneck features are used. ASR is being built. Next, following these three front-end ASR systems, an automatic scoring engine that evaluates non-native English progress is provided to assign pronunciation scores to input utterances of students, and score features are provided. Is extracted, and a score value for the uttered response is estimated.

  Non-Patent Literature 2 states that such a method is evaluated based on a scoring corpus, and a scoring result close to a human evaluator is obtained. In addition, in this method, scoring is performed based on deep learning, so that better results can be obtained as compared with the method using GMM, and furthermore, a tandem type using bottleneck features should be adopted. According to the report, a very high correlation coefficient such as 0.58 at the item level and 0.78 at the pronunciation level was achieved.

M. Nicolao, A. V. Beeston, and T. Hain, "Automatic Assessment of English Learner Pronunciation Using Discriminative Classifiers", in Proceedings of IEEE ICASSP (International Conference on Acoustics, Speech and Signal Processing) 2015, pp. 5351-5355, 2015 J. Tao, S. Ghaffarzadegan, and L. Chen, K. Zechner, "Exploring Deep Learning Architectures for Automatically Grading Non-native Spontaneous Speech", in Proceeding of IEEE ICASSP (International Conference on Acoustics, Speech and Signal Processing) 2016, pp. 6140-6144, 2016

  However, in the conventional techniques as described in Non-Patent Documents 1 and 2, there are some problems because a technique based on forced alignment type phoneme alignment and likelihood score calculation is employed. Has occurred.

  For example, in these conventional techniques, it is necessary to use a high-accuracy acoustic model for a native (a speaker whose native language is the target language), but a large amount of a learning corpus is indispensable to construct this model. In addition, utterance notation is required. Further, for example, when performing forced cut-out type phoneme alignment and likelihood ratio calculation as in Non-Patent Document 1, a non-native acoustic model is also required, and a large amount of non-native corpus is indispensable for construction of this model. Become. However, preparing and using such a large amount of corpus requires a great deal of time and is not practical in terms of cost.

  Further, in the method based on the forced cutout phoneme alignment as described above, the pronunciation data of the teacher corresponding to the evaluation target pronunciation data is required. Furthermore, the spoken text is always fixed and given a priori. Consequently, the methods based on forced segmentation phoneme alignment and likelihood score calculation have low adaptability.For example, when applying to another different language, data related to that language is used. It is necessary to repeat the same process using the same.

  Furthermore, if an ASR based on DNN such as the technique described in Non-Patent Document 2 is used, the system becomes much more complicated. That is, in order to construct an ASR using a highly accurate DNN, an enormous amount of a learning corpus is required.

  When a pronunciation score is given to an ASR output based on non-native speech, it is a major premise that no speech recognition error exists in the ASR. However, such errors can usually arise not only from non-native speech recognition itself, but also from failure to recognize certain phrases in various cases, making it difficult to control. I have.

  Furthermore, the construction of the ASR by the DNN uses a huge amount of speech corpus, so that the amount of arithmetic processing is enormous, and a very large memory is required. In addition, a large number of parameters must be set and adjusted. Will be.

  Therefore, an object of the present invention is to provide a program, an apparatus, and a method capable of evaluating pronunciation of a predetermined language by an evaluation target without relying on a speech recognition technique which has a high processing load due to high accuracy.

According to the present invention, there is provided a pronunciation evaluation program that causes a computer mounted on a device that evaluates pronunciation of a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. Reference score distribution obtaining means for obtaining the obtained score distribution information of the evaluation reference target,
Target score distribution determining means for obtaining a plurality of scores for pronunciation of the predetermined language by the evaluation target using the language identification model, and determining score distribution information of the evaluation target;
Evaluation score determination for determining an evaluation score for the pronunciation of the predetermined language by the evaluation target based on a value related to a distribution parameter in a distribution of a difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. A pronunciation evaluation program that causes a computer to function as a means is provided.

  Specifically, in the pronunciation evaluation program according to the present invention, the evaluation score determination means may calculate a value related to variance as a value related to the distribution parameter, and determine the evaluation score based on the value related to the variance. preferable. Furthermore, it is preferable that the evaluation score determination means calculates a maximum value in the distribution of the difference as a value related to the distribution parameter, and determines the evaluation score based on the maximum value.

  Further, as one embodiment of the pronunciation evaluation program according to the present invention, it is preferable that the evaluation score determination means determines the evaluation score by applying the value related to the distribution parameter to a learned evaluation score estimation model.

Further, as another embodiment of the pronunciation evaluation program according to the present invention, the reference score distribution acquiring means includes a normal distribution parameter representing a histogram generated by a plurality of scores for the pronunciation of the predetermined language by the evaluation reference target. Information that contains the information as score distribution information for the evaluation standard,
It is also preferable that the target score distribution determining means generates a histogram of the plurality of obtained scores, and sets the score distribution information of the evaluation target as information including a normal distribution parameter representing the histogram.

Furthermore, as still another embodiment of the pronunciation evaluation program according to the present invention, the target score distribution determining means newly obtains the score of the evaluation target, updates the score distribution information of the evaluation target,
It is also preferable that the evaluation score determination means updates the evaluation score for the pronunciation of the predetermined language by the evaluation target based on the updated value of the distribution parameter in the distribution of the difference related to the score distribution information of the evaluation target.

  Further, the pronunciation evaluation program according to the present invention obtains a plurality of scores for pronunciations of the predetermined language by the evaluation reference target using the language identification model, determines score distribution information of the evaluation reference target, and obtains a reference score. It is preferable that the computer further functions as a reference score distribution determining unit that outputs to the distribution obtaining unit.

  Further, as a specific example in the pronunciation evaluation program according to the present invention, the evaluation target may be a learner of the predetermined language, and the evaluation reference target may be a plurality of pronunciation providers who speak the predetermined language as a native language. preferable.

According to the present invention, it is also a pronunciation evaluation device that evaluates pronunciation of a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. Reference score distribution obtaining means for obtaining the obtained score distribution information of the evaluation reference target,
Target score distribution determining means for obtaining a plurality of scores for pronunciation of the predetermined language by the evaluation target using the language identification model, and determining score distribution information of the evaluation target;
Evaluation score determination for determining an evaluation score for the pronunciation of the predetermined language by the evaluation target based on a value related to a distribution parameter in a distribution of a difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. And a pronunciation evaluation device having means.

According to the present invention, there is further provided a pronunciation evaluation method in a computer mounted on a device for evaluating pronunciation of a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. The obtained score distribution information of the evaluation reference target is obtained, and the score distribution information of the evaluation target is obtained by using the language identification model to obtain a plurality of the scores for the pronunciation of the predetermined language by the evaluation target. Determining
Determining the evaluation score for the pronunciation of the predetermined language by the evaluation target based on the value of the distribution parameter in the distribution of the difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. A method for evaluating pronunciation is provided.

  ADVANTAGE OF THE INVENTION According to the pronunciation evaluation program, apparatus, and method of the present invention, it is possible to evaluate pronunciation of a predetermined language by an evaluation object without using a speech recognition technology that has a high processing load due to high accuracy.

1 is a schematic diagram illustrating an embodiment of a pronunciation evaluation system including a pronunciation evaluation device according to the present invention. It is a schematic diagram which shows roughly the flow in one Embodiment of the pronunciation evaluation method of this invention performed by a reference | standard score distribution determination part, a target score distribution determination part, and an evaluation score determination part. 5 is a graph for explaining an example (student A) of the pronunciation evaluation method according to the present invention. 5 is a graph for explaining an example (student B) of the pronunciation evaluation method according to the present invention. 5 is a graph for explaining an example (student C) of the pronunciation evaluation method according to the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Pronunciation evaluation system / device]
FIG. 1 is a schematic diagram showing one embodiment of a pronunciation evaluation system including a pronunciation evaluation device according to the present invention.

The pronunciation evaluation system of the present embodiment shown in FIG.
(A) a pronunciation evaluation device 1 according to the present invention;
(B) Includes a pronunciation evaluation device 1 and a server 2 communicatively connected via the Internet or the like. The pronunciation evaluation device 1 of FIG. 1A is a device for evaluating pronunciation in a predetermined language. For example, a terminal that downloads a pronunciation evaluation program according to the present invention, that is, a personal computer (PC), a tablet computer, a smartphone, or the like. And so on.

  The pronunciation evaluation device 1 inputs, via the microphone 107, for example, a pronunciation group of a foreign language (predetermined language) being acquired by a user, for example, a student (for example, a learner of a second language) in a foreign language conversation classroom. For example, an evaluation result for the pronunciation group, for example, a five-stage score described later can be presented to the student (user) via the display 105.

Specifically, the pronunciation evaluation device 1 has the following features:
(A) A score obtained using a “Language IDentification (LID) model” that outputs a score related to the likelihood that the input pronunciation is a pronunciation in a predetermined language, and is an evaluation reference target, for example, A reference score distribution obtaining unit 112 that obtains “score distribution information” of a teacher (evaluation reference target) determined by obtaining a plurality of scores for pronunciation in a predetermined language by a native teacher in a foreign language conversation classroom;
(B) Using the same “LID model”, a plurality of scores for pronunciation of the given language by one student (evaluation target) are acquired, and target score distribution for determining “score distribution information” of the student (evaluation target) A determination unit 113;
(C) On the basis of “value related to distribution parameter” in “difference distribution” between “score distribution information” of this student (evaluation target) and “score distribution information” of teacher (evaluation reference target), And an evaluation score determination unit 114 that determines an evaluation score for pronunciation in a predetermined language by an evaluation target).

  Here, the teacher (evaluation reference target) of the above configuration (A) can be a specific one as a reference, but a plurality of teachers are required in order to secure stability and high level as a reference. It is also preferable to employ a teacher. In addition, since the evaluation target (teacher) is a plurality of pronunciation providers who speak the predetermined language to be taught as their native language, that is, it is also preferable that these evaluation reference targets (teachers) are natives of the predetermined language because they serve as identification criteria by the “LID model”. .

The “difference distribution” of the above configuration (C) will be described in detail later. For example, the normal distribution N ₁ (μ ₁ , σ ₁ ² ) and the normal distribution N ₂ (μ ₂ , σ ₂ ² ) "distribution of the difference" in the normal distribution _{N d (μ d = μ 2} -μ 1, σ d 2 = σ 1 2 + σ 2 2) are set to. Further, the “value related to the distribution parameter” in the “difference distribution” may be a value related to the variance σ _d ² in the “difference distribution”, which will also be described later in detail. More specifically, to a maximum value of the probability density in the "distribution of the difference" ^{(= (2π) -0.5 / σ} d = (2π (σ 1 2 + σ 2 2)) -0.5) are also preferred.

Incidentally, in this case, the maximum value of the probability density (= (2π (σ ₁ ² + σ ₂ ² )) − ^0.5 ) as the “value related to the distribution parameter” of the “difference distribution” is the variance σ ₁ ² and σ ₂ ² Is calculated if is obtained.
(A) Variance σ ₁ ² as “score distribution information” of teacher (evaluation reference target), and (b) Variance σ ₂ ² as “score distribution information” of student (evaluation target)
Is obtained, the student's evaluation score can be finally determined.

  As described above, the pronunciation evaluation device 1 according to the present invention automatically derives the evaluation score for the pronunciation of the student (evaluation target) based on the score obtained using the “LID model”. Here, the “LID model” is a model capable of executing identification and classification of a language type. Specifically, a pronunciation of a student (evaluation target) is inputted, and the pronunciation is a pronunciation of a predetermined language. Output a certain likelihood. That is, there is no model that requires very high accuracy, and therefore, the model is constructed so that a large processing load does not occur in constructing the model.

  Accordingly, the pronunciation evaluation device 1 requires such high accuracy without using a speech recognition model (ASR (Automatic Speech Recognition) model) having a large processing load due to high accuracy, and further, without requiring a non-native corpus. It is possible to provide an evaluation score with sufficiently high accuracy by focusing on “values related to distribution parameters” in “distribution of differences” by utilizing “LID model” which is not used.

  In addition, according to the pronunciation evaluation device 1, it is not necessary to convert the utterance data into text for the pronunciation evaluation, and it is unnecessary to provide a reference utterance sentence by, for example, an evaluation reference target (teacher).

  Therefore, it is also possible to further reduce the amount of processing operations and the required memory in the pronunciation evaluation device 1, and in that case, for example, the pronunciation evaluation device 1 can be housed in a portable terminal having a certain limit in computational power. It is. Further, for example, a mode in which the final pronunciation evaluation score is output substantially in real time can be realized.

  Incidentally, the reference score distribution determination function (reference score distribution determination unit) for determining the “score distribution information” of the teacher (evaluation reference target) in the above configuration (A) is, as shown by reference number 111 in FIG. The device 1 itself may have it, or as a modification, it is preferable that the server 2 is provided with this function (reference score distribution determination unit 212). In the latter case, the “score distribution information” of the teacher (evaluation reference target) is transmitted and provided from the server 2 to the pronunciation evaluation device 1.

  The LID model construction function (language identification model construction unit) for constructing a “LID model” which is a main component of the pronunciation evaluation device 1 is also provided by the pronunciation evaluation device 1 itself as indicated by reference numeral 121 in FIG. It is also preferable that the server 2 has this function (the language identification model construction unit 211). In the latter case, the constructed (learned) “LID model” is transmitted and provided from the server 2 to the pronunciation evaluation device 1. The server 2 also holds the native corpus (reference number 102 in FIG. 1) used at the time of construction, instead of the pronunciation evaluation device 1.

  Furthermore, the evaluation target of the pronunciation evaluation score is, of course, not limited to students (language learners) as in the present embodiment. For example, a dialogue of a predetermined language is enabled by learning (with a dialogue scenario). It may be an automatic dialog system. Naturally, the evaluation criteria are not limited to teachers (language professors). For example, by repeating the update, an utterance equivalent to that of a native speaker can be made (can be sufficiently adopted as a criterion). It can also be a system.

[Configuration of pronunciation evaluation device]
Similarly, according to the functional block diagram of FIG. 1, the pronunciation evaluation device 1 includes a communication interface unit 101, a native corpus 102, a user pronunciation storage unit 103, an evaluation score storage unit 104, and a touch panel display (TP / DP). 105, a microphone (MC) 107, a speaker (SP) 108, and a processor memory.

  Here, the processor memory stores one embodiment of the pronunciation evaluation program according to the present invention, and has a computer function, and executes the pronunciation evaluation program to execute the pronunciation evaluation processing. I do. For this reason, the pronunciation evaluation device 1 may be, for example, a personal computer (PC), a notebook or tablet computer, a smartphone, or the like, on which the pronunciation evaluation program according to the present invention is installed.

  Further, the processor memory includes a reference score distribution determination unit 111 having a language identification unit 111a, a reference score distribution acquisition unit 112, a target score distribution determination unit 113 having a language identification unit 113a, a difference distribution calculation unit 114a, and an evaluation unit. It has an evaluation score determination unit 114 having a score estimation unit 114b, a language identification model construction unit 121, an evaluation score estimation model construction unit 122, a communication control unit 131, and an input / output control unit 132. Note that these functional components can be regarded as functions of the pronunciation evaluation program stored in the processor memory. The flow of processing shown by connecting the functional components of the pronunciation evaluation device 1 in FIG. 1 with arrows is also understood as an embodiment of the pronunciation evaluation method according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the communication control unit 131
(A) receiving the learned LID (language identification) model from the server 2,
(B) receiving the learned evaluation score estimation model from the server 2;
(C) receiving the reference score distribution information that is the teacher (evaluation reference target) score distribution information from the server 2,
(D) It is also preferable to transmit the evaluation score output from the evaluation score determination unit 114 to an external information processing device.

  Note that, in the embodiment in which the pronunciation evaluation device 1 includes the language identification model construction unit 121, the reception of the above-described (a) LID model is unnecessary. Further, in the embodiment in which the pronunciation evaluation device 1 includes the evaluation score estimation model construction unit 122, the reception of the evaluation score estimation model of (b) is unnecessary. Furthermore, in the embodiment in which the pronunciation evaluation device 1 includes the reference score distribution determining unit 111, the reception of the reference score distribution information of (c) is not necessary. Furthermore, when the evaluation score output from the evaluation score determination unit 114 only needs to be presented to the student (evaluation target) via, for example, the display 105, it is unnecessary to transmit the evaluation score of (d) to the outside. Becomes

  The user pronunciation storage unit 103 stores, for example, pronunciation data of a predetermined language by a student (evaluation target) obtained through the microphone 107 and converted by the input / output control unit 132 into digital data of a predetermined format. save. Here, the user pronunciation storage unit 103 stores the pronunciation data in association with the student's identifier (ID) for each student and stores the score distribution information unique to the student for each student using the pronunciation data. Can be generated.

  The language identification unit 111a of the reference score distribution determination unit 111 extracts a plurality of pronunciation data (features generated from) for each teacher in the native teacher group as an evaluation reference target from, for example, the native corpus 102 and converts the data into an LID model. The LID model outputs a certainty score corresponding to the certainty that the pronunciation is a pronunciation in a predetermined language. Here, the pronunciation of the native teacher group may be acquired via the microphone 107, converted into digital pronunciation data by the input / output control unit 132, and used as input data to the LID model.

The reference score distribution determination unit 111
(A) generating a probability score histogram from the plurality of probability scores acquired as described above,
(B) determining a normal distribution representing the generated accuracy score histogram, that is, a normal distribution fitted to the histogram;
(C) the distribution parameter information of the normal distribution, for example, an average mu ₁ and variance sigma ₁ ^2, a confidence score distribution information teacher group (Evaluation reference object) is taken as the "reference score distribution information".

  Here, the certainty score histogram can be a graph in which, for example, the certainty score is set on the horizontal axis, and for each certainty score section, the frequency (count number) of the certainty score corresponding to the section is set on the vertical axis.

  The fitting process for the accuracy score histogram in (b) above can be performed using a known method such as a nonlinear least squares method. For example, a function fitting function in various commercially available data analysis software can be used. Good.

  Further, as a modification, when the above “reference score distribution information” is determined by the reference score distribution determination unit 212 of the server 2, the reference score distribution acquisition unit 112 transmits the “reference score distribution information” via the communication interface unit 101. Information ".

  The language identification model construction unit 121 constructs an LID model used by the reference score distribution determination unit 111 and the target score distribution determination unit 113. Here, the LID model is constructed by a known deep neural network (DNN, Deep Neural Network) algorithm, for example.

  Specifically, a feature is generated from the digital pronunciation data of the native teacher extracted from the native corpus 102, and the feature and the language type of the pronunciation (the native language of the native) are used as learning data. , And DNN, the LID model is constructed.

  Therefore, the LID model is constructed for each predetermined language (for example, for each type of language that the student learns) using the native corpus of the predetermined language. For example, the LID model for English, the LID model for Greek, the Chinese language It is also preferable to prepare in the form of a word LID model. In this case, using these LID models, it is also possible to execute a language classification process for determining which of a plurality of language types the input pronunciation corresponds to.

  Here, unlike the ASR (Automatic Speech Recognition) model that requires high accuracy, the LID model only needs to perform the identification processing of a predetermined language. It can be constructed with learning data. As a result, for example, pronunciation evaluation functions (for various languages) corresponding to various languages can be easily realized.

  As described above, an embodiment in which the server 2 constructs an LID model in the language identification model constructing unit 211 and provides the LID model to the pronunciation evaluation device 1 is also possible. In this case, the language identification model construction unit 121 becomes unnecessary.

  The target score distribution determination unit 113 having the language identification unit 113a extracts the pronunciation data (features generated from the non-native students) to be evaluated from the user pronunciation storage unit 103, for example, and obtains the reference score distribution determination unit. Input is made to the same LID model as used in 111, and a certainty score corresponding to the certainty that this pronunciation is a pronunciation in a predetermined language is output from the LID model.

  Here, the pronunciation of the non-native student is acquired via the microphone 107, converted into digital pronunciation data by the input / output control unit 132, and temporarily stored in the user pronunciation storage unit 103, as input data to the LID model. May be used.

  The pronunciation of the student is preferably, for example, a read-out of a reference utterance sentence based on the pronunciation of the teacher (which has undergone language identification processing in the reference score distribution determination unit 111). Instead of the reference utterance sentence, it may be uttered freely in a conversation, for example.

The target score distribution determination unit 113
(A) generating a probability score histogram from the plurality of probability scores acquired as described above,
(B) determining a normal distribution representing the generated accuracy score histogram, that is, a normal distribution fitted to the histogram;
(C) The distribution parameter information of the normal distribution, for example, the average μ ₂ and the variance σ ₂ ² are set as “target score distribution information” that is the accuracy score distribution information of the student (evaluation target).

  As described above, the target score distribution determination unit 113 uses the same LID model as that used in the reference score distribution determination unit 111, that is, the native model constructed by the native voice (the native corpus 102), to generate a non-native student. Is scored for pronunciation. Therefore, in the present pronunciation evaluation device 1, there is no need to build and use a model for the evaluation target (non-native), and hence a non-native corpus is not required, and the processing load and memory load of the device are reduced. That's it.

  Here, the language identification processing by the LID model in the target score distribution determination unit 113 can be regarded as a processing for distinguishing between native and non-native in a sense. Therefore, for example, when the pronunciation of a non-native student is close to the pronunciation of a reference native teacher, the LID model is liable to cause a “miss” between native / non-native identification, and the identification result tends to fluctuate greatly. I can say.

In other words, as the pronunciation of a non-native student approaches the native pronunciation, in fact, the generated histogram will have a broader distribution around certain accuracy score values, and hence the generated normal distribution variance sigma ₂ ² shows a larger tendency. On the other hand, as the one in which sound students significantly different from a native, histogram shows the relatively concentrated in a small confidence score value Atari, therefore, tends to be smaller the dispersion sigma ₂ ² Generated normal distribution It is.

Here, the variance σ ₁ ² of the normal distribution relating to the native teacher can be regarded as a fixed value serving as a reference and given a priori in the pronunciation evaluation process of the student. Thus, proximity to pronounce between students and teachers, are understood to be reflected in the variance sigma ₂ ² value of the normal distribution of the students.

  Incidentally, the accuracy score histogram (and its normal distribution) of the student (evaluation target) can be initially generated using, for example, about 5 to 10 pieces of audio data. After that, it is preferable that the generated accuracy score histogram (and the distribution parameter of the normal distribution) is sequentially updated by new voice data by the student. Thus, for example, it is possible to grasp the latest (at this stage) student's language proficiency.

  Further, by generating a certainty score histogram corresponding to the predetermined period every predetermined period after the student starts learning, a change in an evaluation score described later for the student, that is, a state of progress of language learning (language learning progress) It is also possible to grasp footprints).

Similarly, in the functional block diagram of FIG. 1, the evaluation score determination unit 114 calculates a value related to a distribution parameter in the “difference distribution” between the score distribution information of the student (evaluation target) and the score distribution information of the teacher (evaluation reference target). For example, based on the average μ _d and the variance σ _d ² , an evaluation score for pronunciation of a predetermined language by a student (evaluation target) is determined.

Specifically, first, the difference distribution calculation unit 114a of the evaluation score determination unit 114 generates a “difference distribution” from the generated student normal distribution and teacher normal distribution. This “difference distribution” is also a normal distribution, and the average μ _d is
(1) μ _d = μ ₂ −μ ₁
Where the variance σ _d ² is
(2) is generated as ^{_{σ d 2 = σ 1 2 +}} σ 2 2 satisfying the distribution.

Here, the peak value (maximum value) p _d of the probability density in this “difference distribution” is given by the following equation (3): p _d = (2π) ^−0.5 / σ _d = (2π) ^−0.5 / (σ ₁ ² + Σ ₂ ² ) ^0.5
≒ 0.4 / (σ ₁ ² + σ ₂ ² ) ^0.5
It is calculated by As seen from this equation (3), the peak value p _d is the standard deviation sigma ₂ of the normal distribution of students, that is, a monotonically decreasing function of the variance sigma ₂ ^2, therefore, non-native students and native teachers It is a good indicator of the closeness of pronunciation.

For example, as the student's variance σ ₂ ² is larger, a smaller peak value _pd is obtained. Therefore, it is determined that the student's pronunciation is closer to the native (teacher's) pronunciation as the obtained peak value _pd is smaller. be able to. Here, hereinafter, the peak value p _d, is determined before the final evaluation score to a "preliminary evaluation score".

Incidentally, the peak value p _d as this preliminary evaluation score, but is also the variance sigma ₁ ² function of the normal distribution of the native teacher, be regarded as variance sigma ₁ ² is a fixed value as described above since it is, consequently the peak value p _d, it is become possible to adopt as an indicator dispersion sigma ₂ ² is reflected directly in the student.

Also, as a preliminary evaluation score, it is also possible to employ a variance sigma ₂ ^2, or a function of the standard deviation sigma ₂ and variance sigma ₂ ² or the standard deviation sigma ₂ is the value of the variance sigma ₂ ^2. However, among them, the peak value p _d is a value that is easier to derive and easier to handle.

Next, the evaluation score estimating unit 114b of the evaluation score determining unit 114 determines the final “evaluation score” for the pronunciation of the predetermined language by the student, using the “preliminary evaluation score” determined as described above. . Specifically, as one embodiment,
(A) peak value p _d (of “difference distribution”) as a preliminary evaluation score;
(B) The evaluation score is determined by using an evaluation score estimation model constructed by a plurality of teacher data, which is a set of the evaluation score by a human evaluator as a correct answer value.

  The evaluation score of the above (b) can be of various formats and standards. For example, "native level" (Agree level), "quasi-native level" (Mildly Agree level), "average level" "(Undecided level)," Non-native level "(Mildly Disagree level), and" Non-native level "(Disagree).

  Further, the evaluation score estimation model construction unit 122 performs a learning process using the teacher data including the data (a) and (b) to construct an evaluation score estimation model. The model constructed here may be a regression model or may be another machine learning model. Incidentally, as described above, an embodiment in which the server 2 constructs an evaluation score estimation model in the evaluation score estimation model construction unit 213 and provides the evaluation score estimation model to the pronunciation evaluation device 1 is also possible. In this case, the evaluation score estimation model construction unit 122 becomes unnecessary.

In addition, as described above, when the target score distribution determining unit 113 updates the score distribution information (distribution parameter information) of the student (evaluation target), the evaluation score determination unit 114 updates the updated student (evaluation target). Based on the distribution parameters (ie, the updated mean μ _d and variance σ _d ² ) in the “difference distribution” according to the score distribution information, the evaluation score for the pronunciation in the predetermined language by the student (evaluation target) is updated. This makes it possible, for example, to capture the student's latest (at this stage) language proficiency.

  Finally, the evaluation score determination unit 114 may present the determined evaluation score (for example, the above-mentioned five-level score) to, for example, a student to be evaluated via the display 105 or the speaker 108, for example. It is also preferable that the evaluation score is stored in the evaluation score storage unit 104 in association with the identifier (ID) of the corresponding student and the evaluation period information. In this case, for example, the past evaluation status of this student and the transition of the evaluation score can be presented.

[Pronunciation evaluation method]
FIG. 2 is a schematic diagram schematically showing a flow in one embodiment of the pronunciation evaluation method of the present invention, which is performed by the reference score distribution determination unit 111, the target score distribution determination unit 113, and the evaluation score determination unit 114.

As shown in FIG. 2, the pronunciation evaluation method of the present embodiment has two modes, an offline mode and an online mode. Among them, in the offline mode, the reference score distribution determination unit 111
(S1) Using the trained LID model, generating a pronunciation accuracy score by a teacher (evaluation reference target),
(S2) Generate a probability score histogram of the teacher (evaluation reference target),
(S3) A normal distribution fitted to the histogram generated in step S2 is determined, and reference score distribution information (mean μ ₁ and variance σ ₁ ² ) is obtained.

As described above, in the pronunciation evaluation method of the present embodiment, the reference score distribution information (mean μ ₁ and variance σ ₁ ² ) serving as a reference for calculating the final evaluation score of a student (evaluation target) is stored in the offline mode. It is prepared in advance.

On the other hand, in the online mode, thus using the previously prepared standard score distribution information (average mu ₁ and variance sigma ₁ ^2), the evaluation of pronunciation learning in the language by the student (evaluated), for example, substantially real-time Can also be implemented. Specifically, the target score distribution determination unit 113
(S4) Using the same learned LID model as in step S1, a pronunciation accuracy score of the student (evaluation target) is generated,
(S5) A probability score histogram of the student (evaluation target) is generated,
(S6) The normal distribution fitted to the histogram generated in step S5 is determined, and the target score distribution information (mean μ ₂ and variance σ ₂ ² ) is obtained.

  Here, in step S4, if a certainty score is generated using the current pronunciation (data) of the student, it is possible to finally evaluate the pronunciation at the current time (generally in real time). Further, if a certainty score is generated by using the pronunciation (data) of the student in the past predetermined period, the pronunciation evaluation for the predetermined period can be finally performed. In this case, by comparing the final evaluation score at the present time with the final evaluation score in the predetermined period, it is also possible to grasp the progress degree of the language proficiency of the student.

Similarly, as shown in FIG. 3, in this online mode, the evaluation score determination unit 114
(S7) Generate a “difference distribution” of the normal distribution determined in steps S3 and S6,
(S8) The peak value p _d (≒ 0.4 / (σ ₁ ² + σ ₂ ² ) ^0.5 ) of the generated “difference distribution” is calculated,
(S9) Based on the calculated peak value p _d (preliminary evaluation score), a final evaluation score, for example, the above five-level score is determined using an evaluation score estimation model.

  As described above, in the pronunciation evaluation method of the present embodiment, in both the offline mode and the online mode, the LID model that does not require such high precision is used, and the “difference” that requires a relatively small amount of processing operation is sufficient. By applying the calculation process of “distribution of”, it is possible to provide an evaluation score of a student (evaluation target) having sufficiently high accuracy. As a result, for example, it is also possible to implement the present pronunciation evaluation method on a portable terminal having a certain limit in calculation ability, and to present the evaluation score of the student (evaluation target) to the student substantially in real time.

[Example]
FIGS. 3, 4 and 5 are graphs for explaining an embodiment of the pronunciation evaluation method according to the present invention. Here, FIG. 3 shows an example for a reference teacher group and student A, FIG. 4 shows an example for student B, and FIG. An example for C is shown.

  First, FIG. 3 (A) shows the accuracy score histogram of English pronunciation by a plurality of native teachers whose native language is English. Here, these accuracy scores are generated using an English LID model. In this histogram, it is understood that the accuracy scores are concentrated around the value “1”. This indicates that most of the pronunciations of the teachers were correctly identified as "English (by native)." However, in the histogram, some accuracy scores take values between “0” and “1”, which indicates that some fluctuation occurs in the present language identification processing.

Next, FIG. 3B shows a normal distribution curve generated by performing fitting processing on the accuracy score histogram of FIG. 3A. From this acquired normal distribution, distribution parameter information (for example, mean μ ₁ and variance σ ₁ ² ) of the teacher group is determined.

  On the other hand, FIG. 3 (C) shows a certainty score histogram of English pronunciation by a student A who is one of the non-natives who is learning English. Here, these certainty scores are generated using the English LID model used for calculating the certainty score in FIG. In this histogram, it is understood that the accuracy scores are concentrated around the value “0”. This indicates that most of Student A's pronunciation was correctly identified as "not (native) English."

  However, in this histogram, some accuracy scores take values close to “1”. In other words, it can be seen that in the language identification processing, it has been determined that some of the voices of the student A are close to native.

Next, FIG. 3D shows a normal distribution curve generated by performing a fitting process on the accuracy score of FIG. 3C. From the acquired normal distribution, distribution parameter information (for example, mean μ ₂ and variance σ ₂ ² ) of student A is determined. Here, the normal distribution curve of the student A shows a broader shape and a larger variance than those of the students B and C described later.

Next, FIG. 3E shows a “difference distribution” (normal distribution) curve between the normal distribution of the student A shown in FIG. 3D and the normal distribution of the teacher group shown in FIG. It is shown. The peak value p _d (≒ 0.4 / (σ ₁ ² + σ ₂ ² ) ^0.5 ) of the “difference distribution” curve is smaller than those of the students B and C described later. This is because, as described above, reflects the variance sigma ₂ ² according to the Student A is relatively large value. In other words, it indicates that the English pronunciation of the student A tends to be determined to be close to the native English pronunciation.

Finally, FIG. 3F shows the evaluation score determined using the evaluation score estimation model, which is a regression model, with the peak value p _d of the student A as the preliminary evaluation score. According to FIG. 3 (F), the evaluation score of the student A is a value between the “native level” (Agree level) and the “quasi-native level” (Mildly Agree level), and the “native level” (Agree level) ).

  Next, the evaluation result of English pronunciation by the student B will be described. First, FIG. 4A shows a probability score histogram of English pronunciation by a student B who is one of the non-natives who is learning English. Here, these certainty scores are also generated by using the English LID model used for calculating the certainty score in FIG. In this histogram, it is understood that almost all accuracy scores are concentrated on the value “0”. This indicates that nearly all of student B's pronunciation was correctly identified as "not (native) English."

Next, FIG. 4B shows a normal distribution curve generated by performing a fitting process on the accuracy score of FIG. 4A. From the obtained normal distribution, distribution parameter information (for example, mean μ ₂ and variance σ ₂ ² ) of student B is determined. Here, the normal distribution curve of the student B shows a sharper shape and a smaller variance than those of the student A and the student C described later.

Next, FIG. 4C shows a “difference distribution” (normal distribution) curve between the normal distribution of the student A shown in FIG. 4B and the normal distribution of the teacher group shown in FIG. It is shown. The peak value p _d (≒ 0.4 / (σ ₁ ² + σ ₂ ² ) ^0.5 ) of the “difference distribution” curve is larger than that of the student A and the student C described later. This is because, as described above, reflects the variance sigma ₂ ² according to the student B is relatively small value. In other words, it indicates that the English pronunciation of the student B is determined to be considerably different from the native English pronunciation.

Finally, FIG. 4 (D) shows the evaluation score determined using the evaluation score estimation model, which is a regression model, with the peak value p _d of the student B as the preliminary evaluation score. According to FIG. 4D, the evaluation score of the student B is a value between the “quasi-non-native level” (Mildly Disagree level and the “non-native level” (Disagree)).

  Next, the evaluation result of English pronunciation by the student C will be described. First, FIG. 5A shows a histogram of accuracy scores of pronunciation of English by a student C who is one of the non-natives who is learning English. Here, these certainty scores are also generated by using the English LID model used for calculating the certainty score in FIG. In this histogram, it is understood that the accuracy scores are concentrated around the value “0”. This indicates that most of the pronunciation of student C was correctly identified as "not (in native) English."

  However, in this histogram, some accuracy scores take values larger than “0”. In other words, it can be seen that in the language identification processing, it has been determined that some of the voices of the student C are slightly closer to native.

Next, FIG. 5B shows a normal distribution curve generated by performing a fitting process on the accuracy score of FIG. 5A. From the obtained normal distribution, distribution parameter information (for example, mean μ ₂ and variance σ ₂ ² ) of the student C is determined. Here, the normal distribution curve of the student C shows a shape having a half width between them, as compared with those of the students A and B, and shows the variance between them.

Next, FIG. 5C shows a “difference distribution” (normal distribution) curve between the normal distribution of the student C shown in FIG. 5B and the normal distribution of the teacher group shown in FIG. It is shown. The peak value p _d (≒ 0.4 / (σ ₁ ² + σ ₂ ² ) ^0.5 ) of the “difference distribution” curve has a magnitude between them when compared with those of the students A and B described above. .

Finally, FIG. 5D shows the evaluation score determined using the evaluation score estimation model, which is a regression model, with the peak value p _d of the student C as the preliminary evaluation score. According to FIG. 5D, the evaluation score of the student C is a value between the “quasi-native level” (Mildly Agree level) and the “average level” (Undecided level), and the “quasi-native level” ( Mildly Agree level).

  As described above in detail, according to the present invention, it is possible to automatically derive an evaluation score for the pronunciation of an evaluation target (for example, a student) based on a score obtained using the LID model. Here, the LID model is a model capable of executing the identification and classification of the language type. Specifically, the pronunciation of the evaluation target (student) is input, and the certainty is a certainty that the pronunciation is a pronunciation of a predetermined language. Is output. That is, very high accuracy is not required, and therefore, the model is constructed so that a large processing load does not occur in constructing the model.

  As a result, according to the present invention, the LID model that does not require such high accuracy is used without using the ASR model that has a high processing load due to high accuracy, and further does not require a non-native corpus, and the “differential It is possible to provide an evaluation score having sufficiently high accuracy by paying attention to the distribution parameter in “distribution”.

  Further, according to the present invention, it is not necessary to convert the utterance data into text for pronunciation evaluation, and it is unnecessary to provide a reference utterance sentence by an evaluation reference target (for example, a teacher). Therefore, it is possible to further reduce the amount of processing calculations and the required amount of memory in the implementation of the present invention. For example, the pronunciation evaluation device according to the present invention can be accommodated in a portable terminal having a certain limit in computational power. Further, for example, a mode in which the pronunciation evaluation score is output substantially in real time can be realized.

  In addition, the present invention makes it possible to evaluate the appropriate language proficiency of individual learners at a lower burden, particularly when providing a language education service in a language school or a public education period. . Further, since the LID model with a low processing load is used, it can be easily applied to pronunciation evaluation in various languages.

  For the above-described various embodiments of the present invention, various changes, modifications, and omissions in the scope of the technical idea and viewpoint of the present invention can be easily performed by those skilled in the art. The foregoing description is merely an example, and is not intended to be limiting. The invention is limited only as defined by the following claims and equivalents thereof.

1 pronunciation evaluation device 101 communication interface unit 102 native corpus 103 user pronunciation storage unit 104 evaluation score storage unit 105 touch panel display (TP / DP)
107 Microphone (MC)
108 Speaker (SP)
111, 212 Reference score distribution determination unit 111a, 113a Language identification unit 112 Reference score distribution acquisition unit 113 Target score distribution determination unit 114 Evaluation score determination unit 114a Difference distribution calculation unit 114b Evaluation score estimation unit 121, 211 Language identification model construction unit 122 , 213 evaluation score estimation model construction unit 131 communication control unit 132 input / output control unit 2 server

Claims (10)

A pronunciation evaluation program that causes a computer mounted on a device that evaluates pronunciation in a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. Reference score distribution obtaining means for obtaining the obtained score distribution information of the evaluation reference target,
Target score distribution determining means for obtaining a plurality of scores for pronunciation of the predetermined language by the evaluation target using the language identification model, and determining score distribution information of the evaluation target;
Evaluation score determination for determining an evaluation score for the pronunciation of the predetermined language by the evaluation target based on a value related to a distribution parameter in a distribution of a difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. A pronunciation evaluation program characterized by causing a computer to function as a means.   2. The pronunciation evaluation according to claim 1, wherein the evaluation score determination unit calculates a value related to variance as a value related to the distribution parameter, and determines the evaluation score based on the value related to the variance. program.   The said evaluation score determination means calculates the maximum value in the distribution of the said difference as a value concerning the said distribution parameter, and determines the said evaluation score based on the said maximum value, The said evaluation score. Pronunciation evaluation program.   4. The evaluation score determination unit according to claim 1, wherein the evaluation score determination unit determines the evaluation score by applying a value related to the distribution parameter to a learned evaluation score estimation model that has been learned. 5. Pronunciation evaluation program. The reference score distribution obtaining means obtains, as score distribution information of the evaluation reference target, information including a distribution parameter of a normal distribution representing a histogram generated by a plurality of the scores for the pronunciation of the predetermined language by the evaluation reference target. And
The target score distribution determining means generates a histogram of a plurality of the obtained scores, and sets the score distribution information of the evaluation target as information including a distribution parameter of a normal distribution representing the histogram. The pronunciation evaluation program according to any one of 1 to 4. The target score distribution determining means newly acquires the score of the evaluation target, updates the score distribution information of the evaluation target,
The evaluation score determination means updates an evaluation score for pronunciation of the predetermined language by the evaluation target based on the updated value of the distribution parameter in the distribution of the difference related to the score distribution information of the evaluation target. The pronunciation evaluation program according to any one of claims 1 to 5, wherein   Using the language identification model, obtain a plurality of scores for the pronunciation of the predetermined language by the evaluation reference target, determine score distribution information of the evaluation reference target, and output the reference score distribution to the reference score distribution obtaining means. The pronunciation evaluation program according to any one of claims 1 to 6, further causing a computer to function as a determination unit.   The said evaluation object is the learner of the said predetermined language, The said evaluation reference object is several pronunciation providers who speak the said predetermined language as a native language, The one of Claim 1 to 7 characterized by the above-mentioned. The pronunciation evaluation program described in. A pronunciation evaluation device that evaluates pronunciation of a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. Reference score distribution obtaining means for obtaining the obtained score distribution information of the evaluation reference target,
Target score distribution determining means for obtaining a plurality of scores for pronunciation of the predetermined language by the evaluation target using the language identification model, and determining score distribution information of the evaluation target;
Evaluation score determination for determining an evaluation score for the pronunciation of the predetermined language by the evaluation target based on a value related to a distribution parameter in a distribution of a difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. Means for evaluating pronunciation. A pronunciation evaluation method in a computer mounted on a device for evaluating pronunciation in a predetermined language by an evaluation target,
A score obtained using a language identification model that outputs a score related to the accuracy of the input pronunciation being the pronunciation in the predetermined language, and determined by obtaining a plurality of scores for the pronunciation in the predetermined language by the evaluation reference target. The obtained score distribution information of the evaluation reference target is obtained, and the score distribution information of the evaluation target is obtained by using the language identification model to obtain a plurality of the scores for the pronunciation of the predetermined language by the evaluation target. Determining
Determining the evaluation score for the pronunciation of the predetermined language by the evaluation target based on the value of the distribution parameter in the distribution of the difference between the score distribution information of the evaluation target and the score distribution information of the evaluation reference target. A pronunciation evaluation method characterized by having.

Images