KR102442020B1 - Method and apparatus for automatic proficiency evaluation of speech - Google Patents

Abstract

A speaking fluency evaluation method of the present invention comprises the steps of: extracting voice feature data from input voice data; obtaining time-aligned character string information by performing forced alignment on the input text data and the extracted speech feature data based on the native speaker and non-native speaker acoustic models; and evaluating the fluency of the input speech data using the time-aligned character string information and a linear regression-based fluency score model.

Description

Method and apparatus for automatic proficiency evaluation of speech

The present invention relates to a technique for evaluating the fluency of automatic speech using speech recognition technology.

In general, a non-native speaker's speech greatly degrades the performance of Automatic Speech Recognition. This is because the existing automatic speech recognition is learned based on the native speaker's voice, and thus does not reflect the various variations of the non-native speaker's voice. Therefore, the automatic fluency evaluation system for the learner's voice based on ASR is inevitably affected by the performance degradation of ASR. After all, it is difficult to accurately evaluate fluency with the existing ASR technology.

SUMMARY OF THE INVENTION An object of the present invention is to provide a system and method for evaluating the fluency of speaking using an acoustic model trained with augmented data in the presence of a predetermined speech script (or speech string) in order to obtain an accurate fluency assessment. is to provide

The above and other objects, advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

According to an aspect of the present invention, there is provided a speaking fluency evaluation method, comprising: extracting voice feature data from input voice data; obtaining time-aligned character string information by performing forced alignment on the input text data and the extracted speech feature data based on the native speaker and non-native speaker acoustic models; and evaluating the fluency of the input speech data using the time-aligned character string information and a linear regression-based fluency score model.

A computing device according to another aspect of the present invention is a computing device including a processor, a memory, a storage, and a system bus connecting them to evaluate speaking fluency, wherein a software module executing on the processor includes: a voice feature extraction module for extracting data; a forced alignment processing module for obtaining time-aligned character string information by performing forced alignment on the input text data and the extracted voice feature data based on the native speaker acoustic model and the non-native speaker acoustic model; and a fluency evaluation module for evaluating the fluency of the input speech data using the time-aligned character string information and a linear regression-based fluency score model.

According to another aspect of the present invention, there is provided a speaking fluency evaluation method, comprising: extracting voice features from an input voice generated by a learner reading a predetermined voice script; recognizing the input voice in string units by performing forced alignment on the input text data and the extracted voice features based on the native speaker acoustic model and the non-native speaker acoustic model; and evaluating the fluency of the input voice by using a result of recognizing the input voice in units of character strings and a fluency score model.

According to the present invention, it is possible to significantly improve the reliability of the speaking evaluation by evaluating the fluency of speaking using the acoustic model learned with the augmented data.

1 is a diagram schematically illustrating an overall process illustrating an automatic fluency evaluation method of a learner (non-native speaker) speaking according to an embodiment of the present invention.
2 is a diagram illustrating a pseudocode according to an algorithm of a modified and changed time/frequency masking technique according to an embodiment of the present invention.
3 is a table showing features based on frequency and length among fluency evaluation features according to an embodiment of the present invention.
4 is a table showing syllable-based features among fluency evaluation features according to an embodiment of the present invention.
5 is a table showing acoustic characteristics among fluency evaluation characteristics according to an embodiment of the present invention.
6 is a table showing the performance of the Pearson correlation coefficient average of the automatic fluency evaluation method according to an embodiment of the present invention.
7 is a diagram illustrating a computing device to which the automatic speaking fluency evaluation method of the present invention can be applied.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

The present invention automatically evaluates the fluency of a non-native speaker's colloquial vocalization (speaking) in the presence of a predetermined vocal script. That is, the automatic fluency evaluation method for colloquial vocalization (speaking) of a non-native speaker according to the present invention is a method of automatically evaluating the fluency of an input voice generated by a foreign language learner (non-native speaker) reading a predetermined spoken sentence (voice script). to be.

1 is a diagram schematically illustrating an overall process illustrating a method for automatic fluency evaluation of learner speaking according to an embodiment of the present invention.

Referring to FIG. 1 , the automatic fluency evaluation method of a learner (non-native speaker) speaking according to an embodiment includes a speech feature extraction step (S110, Speech feature extraction step), a forced-alignment step (S130, forced-alignment step) and linear regression ( linear-regression, LR)-based fluency evaluation step (S150, proficiency evaluation step) is included.

In the speech feature extraction step S110, a process of extracting speech feature data from input speech data generated by a learner reading a speech script is performed. That is, a speech feature for forced-alignment is extracted from the input speech.

In the forced alignment step (S130), the bidirectional long short-term memory (bidirectional long short-term) learned from the voice of a native speaker with respect to the input text data (input text data) corresponding to the spoken script and the voice feature data extracted in the voice feature extraction step (S110) A process of processing forced alignment using a memory: BLSTM) based acoustic model (132, hereinafter referred to as a 'native speaker acoustic model') is performed. At the same time, a process of processing forced alignment using a BLSTM-based acoustic model 134 (hereinafter referred to as a 'non-native speaker acoustic model') trained with a non-native speaker's voice is performed on the input text data and the speech feature data. Here, the native speaker acoustic model 132 (acoustic model for native speech) is learned only from the native speaker's speech corpus. On the other hand, an acoustic model for nonnative speech (134) is trained with augmented data of the non-native speaker corpus and the non-native speaker corpus. For augmentation of voice data, for example, a speed perturbation technique and a time-frequency masking technique are used. The forced sorting process is a process of recognizing the input voice data in string units by forcibly aligning the input text data (voice script) and the input voice data. Through this recognition process, time-aligned string information is obtained. That is, in the forced alignment step (S130), time-aligned string information (hereinafter, first time-aligned) according to the forced alignment based on the native speaker acoustic model with respect to the input text data and the voice feature data string information) and time-aligned string information (hereinafter, second time-aligned string information) according to forced alignment based on a non-native speaker acoustic model are obtained.

The fluency evaluation step S150 includes a feature extraction step for proficiency evaluation (S151) and a fluency score calculation step (S153, proficiency scoring step). In the feature extraction step (S151) for fluency evaluation, the fluency evaluation features including phonetic features and acoustic features are extracted from the first and second time-aligned character string information obtained in the forced alignment step (S130). process is performed. In the fluency score calculation step S153 , a process of calculating a plurality of fluency evaluation scores using a plurality of linear-regression (LR)-based fluency evaluation models is performed. In this embodiment, five fluency evaluation models 153_1, 153_2, 153_3, 153_4, and 153_5 are used, and accordingly, five fluency evaluation scores are calculated. The five fluency evaluation scores are the overall fluency score (S _holistic ), the score for segmental accuracy (S _segment ), the score for phonological accuracy (S _phonology ), the fluency score (S _fluency ), and stress and a score for pitch and accent accuracy (S _pitch ).

Hereinafter, each step ( S110 , S130 , and S150 ) will be described in more detail.

Speech feature extraction step (S110)

As described above, the step of extracting speech features ( S110 ) is a process of extracting speech features from input speech data generated when a learner who wants to learn a foreign language reads a predetermined speech script.

For extracting speech features, for example, a Mel Filter Bank may be used. When a Mel filter bank is used, voice feature data may be extracted through the following steps.

First, input speech data in the time domain is divided into frames in units of 10 ms. Then, after calculating a power spectrum (or frequency) for each frame, a Mel filter bank is applied to the calculated power spectrum (or frequency), and a 40-dimensional Mel Filter Bank ) to extract the value. Next, the 40-order Mel filter bank values of the previous 7 frames and the following 7 frames are concatenated with the extracted 40-th order Mel filter bank values. By doing so, for each frame, speech feature data including 600-order Mel filter bank values (= 40-th order Mel filter bank values + 280-th order Mel filter bank values + 280-th order Mel filter bank values) are extracted .

Forced alignment step (S130)

In the forced alignment step (S130), as described above, forced alignment is performed on the input text data and voice feature data using the BLSTM-based native speaker acoustic model 132 and the BLSTM-based non-native speaker acoustic model 134, respectively. Thus, first and second time-aligned character string information is obtained.

Forced sorting is a process of exactly matching a section corresponding to an utterance string among voice data. First, the speech string is expanded into a phoneme-based search space using a grapheme to pheneme.

The speech feature data extracted from the input speech data is forcibly aligned based on the native speaker acoustic model 132 in the search space extended from the speech string. Simultaneously or sequentially, the speech feature data extracted from the input speech data are forcibly aligned based on the non-native speaker acoustic model 134 with respect to the search space extended from the speech string.

According to the forced alignment performed for each acoustic model, first and second time-aligned character string information is respectively obtained, and each of the first and second time-aligned character string information is (a) an input corresponding to a speech script start/end time per word for each word included in the text data, (b) the start time and end time for each phoneme of each word included in the input text data corresponding to the speech script ( start/end time per phoneme of each word), (c) the acoustic model-based probability value according to the null hypothesis (H0), (d) the acoustic model-based probability value according to the alternative hypothesis (H1) etc.

Hereinafter, the BLSTM-based native speaker acoustic model 132 and the BLSTM-based non-native speaker acoustic model 134 will be described in detail.

The BLSTM-based native speaker acoustic model 132 for forced alignment is used to determine whether input speech data is similar to a native speaker's speech. The BLSTM-based non-native speaker acoustic model 134 for forced alignment is used to accurately acquire temporal information about words and phonemes of input text data.

The BLSTM based native speaker acoustic model 132 includes, for example, 1 input layer, 4 BLSTM layers, 1 fully connected layer, 1 softmax layer, and the like. The BLSTM-based native speaker acoustic model 132 may be another type of deep learning-based acoustic model. The BLSTM-based native speaker acoustic model 132 is learned from native speaker utterances recorded in a quiet environment such as an office environment. As such, when the native speaker acoustic model 132 used to determine the similarity with the native speaker's voice is learned from the recorded voice data or the augmented voice data in a noisy environment, the degree of congestion in the similarity discrimination can be increased to reduce the similarity discrimination performance. have. Therefore, in learning the native speaker acoustic model 132, voice data or augmented data collected in a noisy environment is not used as training data (training data). The BLSTM-based non-native speaker acoustic model 134 may have the same configuration as the BLSTM-based native speaker acoustic model 132 . The BLSTM-based non-native speaker acoustic model 134 also includes 1 input layer, 4 BLSTM layers, 1 fully connected layer, 1 softmax layer, and the like.

The non-native speaker acoustic model 134 is trained from non-native utterances (non-native speaker corpus) recorded in various environments, including office environments and noisy environments. Additionally, the non-native speaker acoustic model 134 used to accurately acquire temporal information (sections of speech strings) for words and phonemes is augmented according to the speech data augmentation technique applied to improve the performance of the deep learning-based acoustic model. It is learned from voice data. For reference, since the deep learning-based acoustic model shows stable performance when trained with a larger amount of voice data compared to the existing GMM-based acoustic model, the augmented voice data is used in various ways.

Hereinafter, speech data augmentation for the BLSTM-based non-native speaker acoustic model 134 will be described in detail.

For augmentation of voice data (or non-native speaker's voice corpus), as described above, a speed perturbation technique and a time/frequency masking technique may be used.

When the speed perturbation technique is used, the original voice data may be converted and augmented at a preset double speed. For example, the original voice data may be converted at 0.9 times the speed according to the speed perturbation technique to generate the augmented speech data and 1.1 times speed to generate the augmented speech data. For this velocity perturbation technique, for example, an SoX tool (audio manipulation tool) may be used. The velocity perturbation technique is described in "Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur, Audio Augmentation for Speech Recognition, Proc. of Annual Conference of the International Speech Communication Association (INTERSPEECH), Sep. 2015, pp. 3586-3589." It is introduced in detail in the literature of A detailed description of the velocity perturbation technique is substituted for the description given in the literature above.

The time/frequency masking technique is described in "Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le, SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition, Proc. . of Annual Conference of the International Speech Communication Association (INTERSPEECH), Sep. 2019.". A detailed description of the time/frequency masking technique is replaced with the description described in the above document. However, in the augmentation of voice data according to an embodiment of the present invention, the time/frequency masking technique introduced in the above document is modified and changed and used.

2 is a diagram illustrating a pseudocode according to an algorithm of a modified and changed time/frequency masking technique according to an embodiment of the present invention.

As shown in FIG. 2 , the main difference from the scheme introduced in this document is that various parameters r _F , r _T , m _F,max , m _{T, max} are assigned to appropriate values in the present invention. Here, the parameter r _F is the maximum masking frequency ratio for the frequency channel, the parameter r _T is the maximum masking frequency ratio for the time step, the parameter m _F,max is the maximum number of frequency channel sections to be masked at one time, and the parameter m _{T, max} are the maximum number of time step sections to be masked at one time. In an embodiment of the present invention, r _F , r _T , m _{F, max} , m _{T and max} may be set to 0.15, 0.15, 2, and 2, respectively.

In augmentation of voice data according to an embodiment of the present invention, both augmentation techniques introduced above may be applied. For example, first, a velocity perturbation technique is performed on the original speech data. Thereafter, a time/frequency masking technique is applied to each of the two types of voice data augmented according to the original voice data and the velocity perturbation technique to obtain additional three types of augmented data. As a result, five types of augmented voice data are obtained for each original voice data.

First, by using the speed perturbation method, the augmented voice data converted by 0.9x speed of the original voice data is generated, and also the augmented voice data converted at 1.1x speed is generated. Next, the augmented voice data is additionally generated by applying the time/frequency-masking method to the original voice data, the voice data augmented at 0.9x speed, and the voice data augmented at the 1.1x speed.

Through this, with respect to one original voice data, time/frequency of 0.9x-speed augmented voice data, 1.1x-speed augmented voice data, time/frequency-masked augmented voice data of original voice data, and 0.9x-speed voice data -Obtain five types of augmented voice data, including masked augmented voice data and time/frequency-masked augmented voice data of 1.1x voice data. The present invention is not limited thereto, and it goes without saying that various methods may be applied to augment the voice data.

fluency Evaluation step (S150)

The fluency evaluation step (S150), as described above, uses (or analyzes) the first and second time-aligned character string information obtained in the forced sorting step (S130) to extract a fluency evaluation feature. Fluency scores (S _{holistic ,} S _segment , and a fluency score calculation step (S153) of calculating S _phonology , S _fluency , and S _pitch ).

One fluency score ( S _t ) is calculated by the following equation using a corresponding linear regression (LR)-based fluency score model ( R _t ).

는 유창성 평가요소(t)에 대한 선형회귀모델의 i 번째 계수(coefficient)이다. b _t 는 유창성 평가 요소(t)에 대한 선형 회귀 모델의 편향(bias)이다. ∥F∥와 f⁽ⁱ⁾는 유창성 평가용 특징 개수와 i 번째 유창성 특징을 각각 의미한다.where s _t is the fluency score. t is one of holistic evaluation, syllable accuracy (segment), phonology, fluency, and stress and accent accuracy as a fluency evaluation factor.

is the i-th coefficient of the linear regression model for the fluency evaluation factor (t). b _t is the bias of the linear regression model for the fluency evaluation factor (t). |||F|| and f ⁽ⁱ⁾ mean the number of features for fluency evaluation and the i-th fluency feature, respectively.

The fluency evaluation features extracted in the feature extraction step S151 for fluency evaluation include features based on the frequency and length of words and phonemes of the input text data, features based on syllables, and acoustic features.

Frequency and length-based features are obtained using the length and number of each token of time-aligned string information. A syllable-based feature is obtained by using the lengths for each vowel, consonant, syllable, initial and final consonant of time-aligned character string information. Acoustic features are obtained using the acoustic model score and length of time-aligned string information. Specific examples of these features are shown in FIGS. 3-5 , respectively.

Linear-regression (LR)-based fluency scoring models (R _t ) are a training set consisting of non-native utterances (non-native speaker corpus) and native speaker utterances (native speaker corpus) of a reading corpus for fluency evaluation. is learned

Two sets of time-aligned character string information are obtained through forced sorting for the 600-order speech features and speech sentences (or word-based strings for speech recognition results) extracted from each speech of the training set.

After that, R _t is learned with the R toolkit by inputting two sets of time-aligned string information for each utterance in the training set and a target score for s _t .

R Core Team, R: A Language and enviromment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria, 2014

Here, the target score for s _t is calculated as the average value described in the following equation. This course is based on five fluency assessment scores: S _holistic , S _segment , S _phonology , S _fluency , and S _pitch . It is applied to train five linear-regression-based fluency score models.

는 i 번째 발화에 대한 t의 발성점수를 의미하고, t는 총체적 점수(S_holistic), 음정 정확도 점수(S_segment), 음운 정확도 점수(S_phonology), 유창성 점수(S_fluency), 피치와 악센트 점수(S_pitch) 중의 어느 하나이고,

는 r번째 전문 평가자가 i 번째 발화에 대하여 평가한 t의 점수를 의미한다. 총체적 점수(S_holistic), 음정 정확도 점수(S_segment), 음운 정확도 점수(S_phonology), 유창성 점수(S_fluency), 피치와 악센트 점수(S_pitch)는, 예를 들면, 4명의 전문 평가자가 비원민 화자의 발성 데이터를 평가한 점수이다. 각 점수는 5점 스케일로 구성된다. 예를 들면, 1점은 매우 나쁨, 2점은 나쁨, 3점은 적당함, 4점은 좋음 그리고 5점은 완벽함을 의미한다.here,

is the vocal score of t for the ith utterance, t is the overall score (S _holistic ), pitch accuracy score (S _segment ), phonology accuracy score (S _phonology ), fluency score (S _fluency ), pitch and accent score (S _pitch ) any one of,

is the score of t evaluated by the rth expert evaluator for the ith utterance. The overall score (S _holistic ), the pitch accuracy score (S _segment ), the phonology score (S _phonology ), the fluency score (S _fluency ), and the pitch and accent score (S _pitch ) were evaluated by, for example, four expert raters. It is a score that evaluates the vocal data of the min-speaker. Each score consists of a 5-point scale. For example, 1 means very bad, 2 means bad, 3 means moderate, 4 means good, and 5 means perfect.

Also, the linear-regression (LR)-based fluency score models (R _t ) may be models trained in advance based on training data including phonetic features and acoustic features. A linear regression analysis method can be used as a method of training a model for fluency score calculation.

)을 학습한다. First, the combined feature information is extracted from each voice data of the training data and prepared as input data of the linear regression model. In addition, a linear regression model (

) to learn

Experiment

In order to evaluate the speaking fluency evaluation method according to an embodiment of the present invention, a Pearson correlation coefficient was used as a measure to compare the target score calculated by Equation 2 and the system score. For each scale, the performance is summarized as the average value for the 10-fold cross validation test.

In order to understand the effect on voice data augmentation used to efficiently train the BLSTM-based acoustic model used in the forced alignment step in the automatic fluency evaluation method according to an embodiment of the present invention, the present applicant has Experiments were conducted with a combination of types.

6 is a table showing the performance of the Pearson correlation coefficient average of the automatic fluency evaluation method according to an embodiment of the present invention.

The table shown in FIG. 6 is a result of comparing the performance of the Pearson correlation coefficient average with four types of combinations depending on whether data augmentation is applied or not.

The four types of combinations are (a) no voice data augmentation, (b) data augmentation applied only to native speaker voices, (c) data augmentation applied only to non-native speaker voices, and (d) all voice data augmentation applied, and the like.

As can be seen from the table shown in FIG. 6, the fluency evaluation method in which only the non-native speaker data augmentation technique is applied is compared with the case where the voice data is not augmented, when only the native speaker's voice data is augmented, and when the augmentation is applied to all data. It was confirmed that the average correlation coefficient was improved by 0.002, 0.005, and 0.003. Here, it can be seen that the data augmentation of the native speaker data increases confusion in determining whether the input voice is similar to the native speaker's voice, so that the performance deteriorates.

7 is a block diagram illustrating a computing device to which the automatic speaking fluency evaluation method according to the present invention is applied.

Referring to FIG. 7 , the computing device to which the automatic speaking fluency evaluation method according to the present invention is applied includes at least one processor 610 , a memory 630 , and a user interface input device 640 connected through a bus 620 . , a user interface output device 650 , a storage 660 , and a network interface 670 .

The processor 610 may be a central processing unit (CPU), a graphics processing unit (GPU), or a combination thereof. The processor 610 may be a semiconductor device that processes instructions stored in the memory 630 or the storage 660 .

Each of the steps S110 , S130 , and S150 shown in FIG. 1 may be implemented in the form of a software module executed by the processor 610 . For example, the voice feature extraction step S110 of FIG. 1 is implemented as a 'voice feature extraction module', the forced alignment step (S130) is implemented as a 'forced alignment processing module', and the fluency evaluation step (S150) is ' It can be implemented as a 'fluency evaluation module'. At this time, the fluency evaluation module includes a feature extraction module for fluency evaluation for implementing the feature extraction step (S151) for fluency evaluation as a software module and a fluency score calculation module for implementing the fluency score calculation step (S153) as a software module can be subdivided.

A software module resides in a storage medium (ie, memory 630 and/or storage 660) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM. You may.

The memory 630 and the storage 660 may include various types of volatile or nonvolatile storage media. For example, the memory 630 may include read only memory (ROM) and random access memory (RAM).

The storage 660 may store a plurality of algorithms for realizing the automatic fluency evaluation method of speaking according to the present invention and an operating system program that provides an execution environment for these algorithms. For example, the storage 660 may include an algorithm in which a velocity perturbation technique is programmed, an algorithm in which a time/frequency masking technique is programmed, an algorithm in which a linear regression analysis technique is programmed, an algorithm for Mel filter bank extraction, a BLSTM-based algorithm, An algorithm in which the learned native speaker acoustic model and the non-native speaker acoustic model are programmed may be stored. In addition, well-known algorithms used for voice recognition, such as voice feature extraction and forced sorting, may be further stored.

An exemplary storage medium is coupled to the processor 610 , the processor 610 can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor 610 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

Example methods of the present disclosure are expressed as a series of operations for clarity of description, but this is not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement a method according to the present disclosure, other steps may be included in addition to the illustrated steps, other steps may be excluded from some steps, or additional other steps may be included except some steps.

Various embodiments of the present disclosure do not list all possible combinations, but are intended to describe representative aspects of the present disclosure, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure includes software or machine-executable instructions (eg, operating system, application, firmware, program, etc.) that cause an operation according to the method of various embodiments to be executed on a device or computer, and such software or and non-transitory computer-readable media in which instructions and the like are stored and executed on a device or computer.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

extracting speech feature data from the input speech data;
Based on a native speaker acoustic model for determining whether the input voice data is similar to a native speaker's voice and a non-native speaker acoustic model for acquiring temporal information about words and phonemes of the input text data, the input text data and the extracted voice features performing forced sorting on the data to obtain time-aligned character string information; and
evaluating the fluency of the input speech data using the time-aligned character string information and a linear regression-based fluency score model;
A method of evaluating speaking fluency, including In claim 1,
The step of extracting the voice feature data,
and extracting the speech feature data from the input speech data generated by a learner reading a speech script corresponding to the input text data. In claim 1,
The time-aligned string information is
Time information including start time and end time for words and phonemes of the input text data, a probability value based on an acoustic model according to a null hypothesis (H0), and an acoustic model according to an alternative hypothesis (H1) A method of evaluating fluency in speaking, comprising a probability value based on In claim 1,
The native speaker acoustic model is
It is a model trained with a native speaker voice corpus,
The non-native speaker acoustic model is
A method for evaluating fluency of speaking, which is a model trained with augmented data of a non-native speaker voice corpus and a non-native speaker voice corpus. delete In claim 1,
The non-native speaker acoustic model is
It is a model trained using the augmented voice data as training data according to the voice data augmentation technique,
The voice data augmentation technique is
A method for evaluating speaking fluency, comprising at least one of a speed perturbation technique and a time/frequency masking technique. In claim 1,
The step of evaluating the fluency comprises:
extracting features for fluency evaluation by analyzing the time-aligned character string information; and
calculating a fluency score for the input speech data using the fluency evaluation feature and the fluency score model
A method of evaluating fluency in speaking, comprising: In claim 7,
The characteristics for the evaluation of fluency are,
a feature based on frequency and length for each token of the time-aligned string information;
a syllable-based feature indicating the length of each vowel, consonant, syllable, initial consonant, and final consonant of the time-aligned character string information; and
Acoustic characteristic representing the probability value based on the acoustic model for each token of the time-aligned character string information
A method of evaluating fluency in speaking, comprising: In claim 8,
Calculating the fluency score comprises:
Using the above fluency score model, the following formula,

calculating the fluency score according to
where s _t is the fluency score, t is one of holistic, syllable accuracy (segment), phonology, fluency, and stress and accent accuracy as a fluency evaluation factor,

is the i-th coefficient of the linear regression model for the fluency evaluation factor (t), b _t is the bias of the linear regression model for the fluency evaluation factor (t), and f ⁽ⁱ⁾ is the number of features for fluency evaluation and the i-th fluency feature, respectively. In claim 1,
The linear regression-based fluency score model is,
A method for evaluating fluency of speaking, which is learned with a learning set including a non-native speaker corpus and a native speaker corpus according to a linear regression analysis method. In a computing device comprising a processor, memory, storage, and a system bus connecting them, for the assessment of fluency in speaking,
A software module running on the processor,
a voice feature extraction module for extracting voice feature data from the input voice data;
Based on a native speaker acoustic model for determining whether the input voice data is similar to a native speaker's voice and a non-native speaker acoustic model for acquiring temporal information about words and phonemes of the input text data, the input text data and the extracted voice features a forced sort processing module for obtaining time-aligned character string information by performing forced sorting on the data; and
a fluency evaluation module that evaluates the fluency of the input speech data using the time-aligned character string information and a linear regression-based fluency score model;
Computing device comprising. In claim 11,
The voice feature extraction module,
and extracting the speech feature data from the input speech data generated when a learner reads a speech script corresponding to the input text data. In claim 11,
The forced sort processing module,
Time information including start time and end time for words and phonemes of the input text data, a probability value based on an acoustic model according to a null hypothesis, and a probability value based on an acoustic model according to an alternative hypothesis. A computing device for obtaining string information. In claim 11,
The native speaker acoustic model is
It is a model trained with a native speaker voice corpus,
The non-native speaker acoustic model is
A computing device that is a model trained with augmented data of a non-native speaker corpus and a non-native speaker corpus. 15. In claim 14,
The non-native speaker acoustic model is
It is a model trained using the augmented voice data as training data according to the voice data augmentation technique,
The voice data augmentation technique is
A computing device comprising at least one of a speed perturbation technique and a time/frequency masking technique. In claim 11,
The fluency evaluation module,
a feature extraction module for fluency evaluation that analyzes the time-aligned character string information to extract features for fluency evaluation; and
A fluency score calculation module for calculating a fluency score for the input speech data using the fluency evaluation feature and the fluency score model
A computing device comprising a. extracting speech features from input speech generated by a learner reading a predetermined speech script;
Based on a native speaker acoustic model for determining whether the input voice is similar to a native speaker's voice and a non-native speaker acoustic model for acquiring temporal information about words and phonemes of the input text data, the input text data and the extracted voice features recognizing the input voice in string units by performing forced sorting; and
evaluating the fluency of the input voice using a result of recognizing the input voice in character string units and a fluency score model
A method of evaluating speaking fluency, including In claim 17,
The step of extracting the voice feature,
and extracting the extracted Mel filter bank value as the voice feature by applying a Mel filter bank to the power spectrum for the input voice. In claim 17,
In the step of recognizing the input voice in string units,
The non-native speaker acoustic model is
A method for evaluating speaking fluency, which is a model trained using a non-native speaker voice corpus recorded in an environment including an office environment and a noisy environment as training data. In claim 17,
Each of the native speaker acoustic model and the non-native speaker acoustic model,
A method for evaluating the fluency of speaking, which is an acoustic model based on bidirectional long short-term memory.

Images