KR20190012419A - System and method for evaluating speech fluency automatically - Google Patents

Abstract

According to the present invention, a system for automatically evaluating speech fluency includes: an utterance receiving module for receiving an utterance signal of a learner; a memory for storing a program for automatically evaluating the utterance fluency based on the learner′s utterance signal; and a processor for executing the program stored in the memory. The processor is configured to extract the learner′s utterance signal for a target sentence and a phoneme string of a reference utterance signal corresponding to the sentence when the learner′s utterance signal for the target sentence to be evaluated is received by executing the program, and extract loudness information for each vowel from the learner′s utterance signal and the reference utterance signal to calculate a loudness score for the learner′s utterance signal from the loudness information based on an automatic evaluation model learned in advance. The automatic evaluation model aligns the phoneme string of the learner′s utterance signal with the phoneme string of the reference utterance signal based on a dynamic time wrapping (DTW) technique, and detects a phoneme judged to be different from the reference speech signal, extracts the aligned phoneme string and the accent pattern reflecting the detected phoneme, and calculates the accent score thereof.

Description

FIELD OF THE INVENTION [0001] The present invention relates to a system and method for automatically evaluating spoken fluency,

The present invention relates to a system and method for automatically assessing the fluency of a learner's utterance.

In the case of role-play learning or dialog learning, which is one of the most used methods for learning a second foreign language, learning proceeds by listening and following the presented sentences.

Then, after the learner's utterance of each sentence is completed, the information such as the accent, intonation, and pronunciation of the utterance sentence, as well as the recognition result of the uttered sentence, are provided as feedback to the learner in comparison with the utterance of the native speaker.

However, there may be phonemes different from the spoken phonemes of the native speaker according to the pronunciation of the learner, and in such a case, there is a problem that the stress score can not be calculated through the pattern comparison.

In addition, although the rhythm information is one of the important features for evaluating the degree of fluency of learners, there is a problem that such rhythm information is not utilized.

In this regard, Korean Patent Laid-Open Publication No. 10-2015-0015126 (titled " foreign language fluency evaluation system and method ") provides word level feedback including one of pronunciation, Discloses a technology for allowing foreign language learning to be performed by evaluating and providing fluency based on the above-described information.

In the embodiment of the present invention, in performing speaking learning of a second foreign language through role-play learning or dialog learning, which is one method used for learning a second foreign language, Is calculated based on DTW (Dynamic Time Wrapping), thereby providing a system and a method that can automatically evaluate the fluency of speech even when the phoneme string of the sentence uttered by the learner is different from the phoneme string of the presented sentence correctly.

It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to a first aspect of the present invention, there is provided a system for automatically evaluating fluency of utterance, comprising a utterance receiving module for receiving a utterance signal of a learner, And a processor for executing a program stored in the memory, wherein the processor, when receiving the learner's utterance signal for the sentence to be evaluated as the program is executed, Extracting the learner's utterance signal and the phoneme string of the reference utterance signal corresponding to the sentence, extracting the lecture information for each vowel from the learner's utterance signal and the reference utterance signal, From the loudness information, And calculates the score. In this case, the automatic evaluation model aligns the phoneme string of the learner's utterance signal with respect to the sentence and the phoneme string of the reference utterance signal based on a DTW (Dynamic Time Wrapping) technique, and generates a phoneme determined to be different from the reference utterance signal Extracts an accent pattern reflecting the aligned phonemes and the detected phonemes, and calculates the accent score thereof.

The processor can extract the emphasis information for each vowel based on information of energy level, pitch height, and phoneme length from the learner's utterance signal and reference utterance signal.

Wherein the processor assigns a value other than the reference stress value in the case of the phonemes that have been changed among the aligned phoneme strings and a value of the reference stress value that is different from the reference utterance value on the basis of the phoneme having the reference stress value and the phoneme having a value other than the reference stress value And a negative stress score is assigned to the detected phoneme to calculate a stress score corresponding to the stress pattern.

The accent score may include an accent score in units of the sentence and an accent score in units of words included in the sentence.

The processor calculates the stress score by calculating a pattern for a content word in the case of an accent score in units of the word and comparing the patterns with each other and, in the case of the accent score using the sentence as a unit, Pattern and its accent score can be calculated.

The processor can identify the beginning and end of the sentence by adding a phoneme symbol to each of the phonemes located at the beginning of the sentence and the phoneme located at the end of the sentence.

The processor may extract rhythm information using one or more phonemes that are emphasized in the aligned phonemes and time information of one or more phonemes receiving the accent, and may calculate the rhythm score by modeling the rhythm information.

The processor may extract the rhythm information by calculating a standard deviation of an average time interval and a time interval between phonemes receiving the stress with the time information.

Wherein the processor extracts phoneme strings corresponding to the training utterance signal for the automatic evaluation model, extracts the stress information for each vowel for the training utterance signal with the feature information, Matching information can be trained.

A method for automatically evaluating the fluency of utterance in the automatic utterance evaluation system of speech utterance according to the second aspect of the present invention includes: receiving a utterance signal of a learner on a sentence to be evaluated; Extracting a speech signal of the learner and a phoneme string of a reference speech signal corresponding to the sentence; Extracting the loudspeaker information for each vowel from the learner's utterance signal and the reference utterance signal; And a step of calculating an accent score for the learner's utterance signal from the lecture information based on the automatically learned evaluation model, wherein the automatic evaluation model is based on a DTW (Dynamic Time Wrapping) A phoneme string of the learner's utterance signal and a phoneme string of the reference utterance signal are aligned and a phoneme judged to be different from the reference utterance signal is detected and an accent pattern reflecting the aligned phoneme string and the detected phoneme is extracted, And calculates its accent score.

According to any one of the above-mentioned tasks of the present invention, even when the learner's utterance phoneme string is different from the spoken phoneme string of the native speaker, the learner's speaking fluency can be evaluated by calculating the stress score and the rhythm score for the utterance.

1 is a block diagram of a spontaneous fluency automatic evaluation system according to an embodiment of the present invention.
2 is a flowchart of a method for automatically evaluating a speaking fluency according to an embodiment of the present invention.
3 is a flowchart of a method of training an automatic evaluation model.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly explain the present invention in the drawings, parts not related to the description are omitted.

Whenever a component is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements, not the exclusion of any other element, unless the context clearly dictates otherwise.

1 is a block diagram of a spoken fluency automatic assessment system 100 in accordance with an embodiment of the present invention.

The system 100 for automatically evaluating fluency of a learner's utterance according to an embodiment of the present invention includes an utterance reception module 110, a memory 120, a processor 130, and a communication module 140. [

The speech receiving module 110 is configured to receive a learner's utterance signal, for example, a microphone.

The memory 120 stores a program for automatically evaluating the fluency of the utterance based on the learner's utterance signal.

At this time, the memory 120 is collectively referred to as a non-volatile storage device and a volatile storage device which keep the stored information even when power is not supplied.

For example, the memory 120 may be a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD) A magnetic computer storage device such as a NAND flash memory, a hard disk drive (HDD) and the like, and an optical disc drive such as a CD-ROM, a DVD-ROM, etc. .

The processor 130 executes the program stored in the memory 120. [

Specifically, when the learner's utterance signal is received from the utterance reception module 110, the processor 130 can evaluate the degree of fluency of utterance with respect to the utterance signal of the learner through comparison analysis with the reference utterance signal.

In this case, an embodiment of the present invention is for evaluating fluency of utterance to a second foreign language other than the learner's native language, and the reference utterance signal corresponds to a second foreign language utterance of the native speaker on the sentence corresponding to the utterance signal of the learner do. However, an embodiment of the present invention is not necessarily limited to a second foreign language, and may be applied to a native language.

Meanwhile, the automatic utterability evaluation system 100 according to an exemplary embodiment of the present invention may be implemented as a single server and loaded into a server program as a server program, or may be provided as a general PC or a smart terminal It may be installed and serviced in the form of an application.

At this time, when the service is provided in the form of an application, the communication module 140 can receive and use the reference ignition signal, the learned automatic evaluation model, and the evaluation ignition signal from the management server (not shown).

Such a communication module may include both a wired communication module and a wireless communication module. The wired communication module may be implemented by a power line communication device, a telephone line communication device, a cable home (MoCA), an Ethernet, an IEEE1294, an integrated wired home network, and an RS-485 control device. In addition, the wireless communication module can be implemented with a wireless LAN (WLAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60 GHz WPAN, Binary-CDMA, wireless USB technology and wireless HDMI technology.

1 may be implemented in hardware such as software or an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and may perform predetermined roles can do.

However, 'components' are not meant to be limited to software or hardware, and each component may be configured to reside on an addressable storage medium and configured to play one or more processors.

Thus, by way of example, an element may comprise components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, Routines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The components and functions provided within those components may be combined into a smaller number of components or further separated into additional components.

Hereinafter, a method for automatically evaluating the fluency of a learner's utterance in a spoken fluency automatic assessment system 100 according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3. FIG.

2 is a flowchart of a method for automatically evaluating a speaking fluency according to an embodiment of the present invention. 3 is a flowchart of a method of training an automatic evaluation model.

Referring to FIG. 2, the processor 130 executes a program stored in the memory 120, and receives a learner's utterance signal for one or more sentences to be evaluated from the utterance reception module 110 (S110).

Next, the processor 130 extracts the learner's utterance signal, the phoneme string and the time alignment information of the reference utterance signal corresponding to the sentence (S120), and extracts the accent information for each vowel from the learner's utterance signal and the reference utterance signal (S130).

At this time, the processor 130 calculates the learner's utterance signal and the reference utterance signal based on at least one of the learner's utterance signal and the reference utterance signal energy, the pitch, and the phoneme duration, Can be extracted as the bullish information for.

Then, the processor 130 calculates a stress score for the learner's utterance signal from the stress information based on the previously-learned automatic evaluation model (S140).

At this time, the processor 130 aligns the phoneme string of the learner's utterance signal with the phoneme string of the reference utterance signal based on the DTW (Dynamic Time Wrapping) technique by using the previously learned automatic evaluation model. In order to set the start point and the end point in performing the DTW technique, the processor 130 adds a phoneme symbol to the front of the phoneme located at the beginning of the sentence and the phoneme located at the end of the sentence, You can set the beginning and end of the file.

For example, processor 130 may mark the beginning and end of a sentence by adding 'sS' at the beginning of the sentence start phoneme and a phoneme 'sE' at the end of the sentence start end phoneme.

The processor 130 then detects the recognized phoneme by pronouncing it differently from the native speaker's utterance. For this, the processor 130 detects phonemes of the aligned learner and phonemes of the reference speech signal that are different from the reference speech signal, that is, phonemes determined to be erroneous phonemes.

Then, the processor 130 extracts the accent pattern by reflecting the aligned phoneme string and the detected phoneme, and calculates the accent score corresponding to the accent pattern.

For this purpose, the processor 130 assigns values other than the reference accent values, for example, 0, 1, and 2, to the phoneme of the aligned learner and the phoneme string of the reference utterance signal. A phoneme judged to be different from the reference speech signal is detected based on the phoneme having the reference stress value and the phoneme having the value other than the reference stress value, and a positive (-) stress score is given to the detected phoneme, Can be calculated.

In this case, the accent score may include an accent score in units of sentences and an accent score in units of words included in the sentence.

In the case of an accent score based on a word, a pattern for the content word can be calculated and compared with the calculated score. In the case of the accent score in units of sentences, the accent pattern of the whole sentence can be extracted and the accent score thereof can be calculated.

On the other hand, as shown in FIG. 3, the automatic evaluation model extracts a training sound spoken signal (S210), extracts a training sound spoken signal (S220), and outputs the training sound spoken signal The vowel information for the vowel may be extracted (S230), and then the vowel information may be matched to the predetermined evaluation vowel score D1 (S240).

Referring again to FIG. 2, the automatic utterability evaluation system 100 according to an exemplary embodiment of the present invention can evaluate the degree of fluency of utterance using rhythm information about a learner's utterance.

Specifically, the processor 130 extracts rhythm information using one or more phonemes and one or more phonemes that are emphasized in the phoneme string of the aligned learner and the phoneme string of the reference utterance, , And the rhythm score can be calculated by modeling the rhythm information (S160).

Here, the rhythm information is obtained by calculating the average time interval (rhythm average) and the standard deviation (rhythm standard deviation) of the time intervals between the phonemes that are accentuated, and the processor 130 models them, Rhythm score can be calculated.

Meanwhile, in the above description, steps S110 to S240 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed. In addition, even if other contents are omitted, the contents described in the automatic utterance evaluation system 100 of FIG. 1 can also be applied to the automatic utterance evaluation method of FIG. 2 and FIG.

According to the embodiment of the present invention, even when the learner's utterance phoneme string is different from the spoken phoneme string of the native speaker, the learner's speaking fluency can be evaluated by calculating the stress score and the rhythm score for the utterance.

On the other hand, an embodiment of the present invention may also be embodied in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: Speech Fluency Automatic Evaluation System
110: ignition reception module
120: Memory
130: Processor
140: Communication module

Claims (1)

A system for automatically evaluating fluency of speech,
An ignition reception module for receiving a learner's ignition signal,
A memory for storing a program for automatically evaluating the fluency of speech based on the learner's utterance signal;
And a processor for executing a program stored in the memory,
Wherein the processor extracts a learner's utterance signal and a phoneme string of a reference utterance signal corresponding to the sentence when a learner's utterance signal for a sentence to be evaluated is received as the program is executed, And extracting the stress information for each vowel from the reference spoken signal, and calculating a stress score for the spoken signal of the learner from the stress information based on a previously learned automatic evaluation model,
Wherein the automatic evaluation model includes a phoneme string of the learner's utterance signal and a phoneme string of the reference utterance signal with respect to the sentence based on a DTW (Dynamic Time Wrapping) technique, and detects a phoneme determined to be different from the reference utterance signal Extracting an accent pattern reflecting the aligned phoneme string and the detected phoneme, and calculating the accent score thereof.

Images