KR20220036239A - Pronunciation evaluation system based on deep learning - Google Patents

Abstract

The present invention relates to a pronunciation evaluation system.
The pronunciation evaluation system according to the present invention includes a character providing unit that provides sentences necessary for learning, a spoken voice collection unit that collects the voice uttered by the learner in response to the provided sentence, and converts the spoken voice into text form. A text conversion unit that acquires the obtained text into an utterance, an evaluation unit that calculates the matching rate for the sentence by comparing and analyzing the obtained utterance and the provided sentence, and recognizes words that the learner consistently pronounces incorrectly using the evaluated results. A misrecognition word collection unit that collects information about words in which errors occur and stores the collected words in table form, and when the utterance and the provided sentence match, the evaluation results are quantified, and pronunciation accuracy and stress matching are provided. It is provided in color-coded form, and if there is a discrepancy between the utterance and the provided sentence, it includes a control unit that transmits a signal requesting re-utterance for the provided sentence.

Description

Pronunciation evaluation system based on deep learning}

The present invention relates to a pronunciation evaluation system, and more specifically, a pronunciation evaluation that utilizes deep learning and NLP (natural language processing) technology to evaluate the user's pronunciation by matching the provided sentences with the voice information obtained as the user speaks. It's about the system.

Recently, the number of users using computers to learn foreign languages such as English has been increasing. In particular, programs for learning English pronunciation are increasing. When a user utters a specific word or sentence through a microphone, the utterance is analyzed and an evaluation of the user's pronunciation is provided. At this time, voice recognition technology is applied to find out the content of the user's speech, and as a result of the evaluation, feedback appropriate to the score or evaluation level is provided to the user.

In the case of sentence pronunciation learning, the content of the pronunciation learning results provided to the user often indicates only the overall pronunciation accuracy (overall score) for the entire utterance content, and in the case of word pronunciation learning, it indicates whether the word was pronounced correctly. In many cases, it only indicates that it is not. At this time, when uttering multiple words such as sentences, there is a problem in that accurate pronunciation learning results are not provided because some words with problematic pronunciation are not pointed out.

On the other hand, there are cases where the voice recognition results are further utilized to point out and inform the user of parts that the user has mispronounced, but in this case, parts that are prone to errors in Koreans' English pronunciation are built with knowledge in advance and the incorrect pronunciation is recognized through voice recognition. There is a problem that correcting pronunciation is possible only by building additional information about accurate pronunciation by informing the user of this.

In addition to the accuracy of the pronunciation itself, there are cases where suprasegmental evaluation factors are included to measure the naturalness of speech, especially when uttering sentences. It's difficult to point out something and provide details about how it went wrong. Here, suprasegmental evaluation elements are non-segmental items such as sentence intonation, stress, speaking speed, etc., and segmental elements are separable items such as sentences, phrases, syllables, words, and phonemes.

Therefore, there is a need for a foreign language pronunciation evaluation technology that can provide pronunciation learning results for sentences by evaluating suprasegmental elements for each predetermined unit of words.

The technology behind the present invention is disclosed in Republic of Korea Patent Publication No. 10-2019-0068841 (published on June 19, 2019).

The technical problem to be achieved by the present invention is to provide a pronunciation evaluation system that evaluates the user's pronunciation by matching the speech information obtained as the user speaks with the provided sentence using deep learning and NLP (natural language processing) technology. .

In the pronunciation evaluation system according to an embodiment of the present invention for achieving this technical problem, a character providing unit providing sentences necessary for learning, a speech voice collection unit collecting speech uttered by the learner in response to the provided sentences, the above A text conversion unit that converts the spoken voice into text form and obtains the converted text as an utterance, an evaluation unit that calculates a matching rate for the sentence by comparing and analyzing the obtained utterance and the provided sentence, and uses the evaluated results. A misrecognition word collection unit collects information on words that learners consistently pronounce incorrectly and words in which recognition errors occur, stores the collected words in a table, and evaluates when the utterance and the provided sentence match. It includes a control unit that quantifies the results, provides them by color-coded according to pronunciation accuracy and stress matching, and transmits a signal requesting re-utterance for the provided sentence if there is a discrepancy between the utterance and the provided sentence.

The text conversion unit classifies the spoken voice by word, generates an utterance obtained using the classified word, and a plurality of candidate sentences similar to the utterance, and provides time information corresponding to each classified word. can do.

The evaluation unit may compare the provided sentence and utterance word by word and calculate a matching rate according to the number of matched words.

The evaluation unit may classify the provided sentences and utterances word by word, compare each classified word, assign a weight to the provided sentences or utterances, and calculate a matching rate.

The evaluation unit may assign weight to the utterance if the word is pronounced correctly or the word is pronounced once again, and may assign weight to the provided sentence if the word is pronounced incorrectly or the word is omitted.

The evaluation unit may classify the provided sentences and utterances word by word, set the classified words to a plurality of cases, and then calculate a matching rate using the set number of cases.

The misrecognized word collection unit extracts numbers, infrequently used words, and words that learners repeatedly pronounce incorrectly, classifies them as misrecognized words, and stores the classified misrecognized words in a table.

The control unit acquires a waveform using the learner's uttered voice, acquires the stress of the word by matching the acquired waveform with time information of the word included in the utterance, and combines the acquired stress with the stress of the original word. By comparing, you can print out whether the answer is correct or not, expressed in color.

In this way, according to the present invention, based on deep learning and natural language processing (NLP) technology, the user's level of speech for the sentence is evaluated, and the evaluated results are indexed and provided, so that the user can feel a sense of accomplishment and learn English on their own. By entering the pronunciation of , the location of the incorrect pronunciation is pointed out in units of phonemes or syllables and notified to the user. At the same time, detailed analysis of pronunciation accuracy, stress, intonation, speed, etc. can be performed, thereby improving learning efficiency.

In addition, according to the present invention, the recognition rate of the user's pronunciation can be increased by monitoring words that the user repeatedly pronounces incorrectly and continuously updating the conversion table, and classifying them into stressed words and unstressed words. By doing so, the matching rate between the uttered sentence and the provided sentence can be increased.

1 is a configuration diagram for explaining a pronunciation evaluation system according to an embodiment of the present invention.
Figure 2 is a flowchart showing a method of evaluating pronunciation using a pronunciation evaluation system according to an embodiment of the present invention.
Figure 3 is a diagram showing step S210 shown in Figure 2.
FIG. 4 is a diagram for explaining step S220 shown in FIG. 2.
Figure 5 is an example diagram showing the state of converting and outputting a speech sentence in step S240 shown in Figure 2.
FIG. 6 is a diagram illustrating a method of calculating the matching rate by the third method in step S250 shown in FIG. 2.
Figure 7 is an example diagram showing a state in which the results derived in step S270 shown in Figure 2 are output.
FIG. 8 is a diagram for explaining a method of determining the stress of a word in step S270 shown in FIG. 2.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawings may be exaggerated for clarity and convenience of explanation.

Additionally, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

Hereinafter, the pronunciation evaluation system according to an embodiment of the present invention will be described in more detail.

1 is a configuration diagram illustrating a deep learning-based pronunciation evaluation system according to an embodiment of the present invention.

As shown in FIG. 1, the pronunciation evaluation system 100 according to an embodiment of the present invention includes a text providing unit 110, a spoken voice collection unit 120, a text conversion unit 130, an evaluation unit 140, It includes a misrecognition word collection unit 150 and a control unit 160.

First, the text provider 110 provides sentences necessary for learning. The sentences provided at this time may be lines from movies or dramas, conversations needed in daily life, or music lyrics.

The spoken voice collection unit 120 collects the voice spoken by the learner in response to the provided sentence. The learner turns on the microphone function included in the terminal. Then, by reading the text provided into the microphone, the learner's terminal collects the voice spoken by the learner.

The text conversion unit 130 converts the collected spoken voices into text form and obtains them.

The evaluation unit 140 obtains a matching rate by comparing the utterance with the sentences provided for learning. The evaluation unit 140 builds a deep learning model and inputs the utterance and sentences provided for learning into the built deep learning model. Then, the deep learning model classifies the utterance and the sentences provided for learning by word, and then matches the classified words with each other to obtain a matching rate.

The misrecognition word collection unit 150 uses the evaluation results to collect information on words that the learner consistently pronounces incorrectly and words in which recognition errors occur, and stores the collected words in a table.

Finally, if the utterance and the provided sentence match, the control unit 160 quantifies the evaluation results and provides them by color-coding them according to pronunciation accuracy and stress matching. If the utterance and the provided sentence do not match, the provided sentence is provided. A re-ignition request signal is transmitted.

Hereinafter, a method for evaluating pronunciation using a pronunciation evaluation system according to an embodiment of the present invention will be described in more detail using FIGS. 2 to 8.

Figure 2 is a flowchart showing a method of evaluating pronunciation using a pronunciation evaluation system according to an embodiment of the present invention, Figure 3 is a diagram showing step S210 shown in Figure 2, and Figure 4 is a diagram showing step S220 shown in Figure 2. It is a diagram for explaining the steps, and Figure 5 is an example diagram showing the state of converting and outputting an utterance in step S240 shown in Figure 2, and Figure 6 is a matching rate by the third method in step S250 shown in Figure 2. It is a diagram for explaining a method of calculating, and Figure 7 is an example diagram showing the output state of the result derived in step S270 shown in Figure 2, and Figure 8 shows the stress of the word in step S270 shown in Figure 2. This is a drawing to explain how to make a judgment.

First, the learner downloads an application that can evaluate pronunciation and activates it. Then, as shown in FIG. 3, the text provider 110 provides a learnable sentence (S210).

Learners can select any one sentence from among the sentences provided, or they can select them sequentially starting from the first sentence.

When the sentence selection is completed, the control unit 160 receives information about the gender of the learner selected by the learner and the accent for the specific country to be evaluated (S220).

As shown in Figure 4, the present invention allows evaluation to be made corresponding to the learner's selected language method, that is, British or American.

Next, when the learner records the provided sentence, the speech voice collection unit 120 receives the learner's speech voice generated during the recording (S230).

The text conversion unit 130 converts the received spoken voice into text form (S240).

And, as shown in Figure 5, the converted utterance is immediately output through the application.

To elaborate, the text conversion unit 130 classifies the uttered voice by word and obtains an utterance using the classified words. At this time, the text conversion unit 130 generates a plurality of candidate sentences in addition to the utterance sentence.

For example, if the provided sentence is “I am hungry,” the text conversion unit 130 converts the voice spoken in response to the provided text into “I am hungry,” and in addition, “I was hungry,” “I Generate candidate sentences such as “am angry.”

When step S250 is completed, the evaluation unit 140 compares the utterance and the provided sentence to calculate a matching rate (S250).

The evaluation unit 140 calculates the matching rate using one of three methods.

First, the first method will be described. The evaluation unit 140 compares the provided sentence and utterance word by word and calculates the matching rate according to the number of matched words.

For example, assuming that the provided sentence is “I want to go home” and the utterance is “I want to go home,” the evaluation unit 140 calculates a matching rate of 100% for the corresponding utterance. .

On the other hand, when the utterance is “I go home,” the evaluation unit 140 calculates a matching rate of 20% for the corresponding utterance.

Next, explaining the second invention, the evaluation unit 140 classifies the provided sentences and utterances by word, compares each classified word, assigns a weight to the provided sentences or utterances, and calculates a matching rate.

For example, the provided sentence was "I want to go home", which was classified into words as "I", "want", "to", "go", and "home", and the utterance corresponding to the provided sentence was Assume that “I go home” is classified into words as “I”, “go”, and “home”.

Then, the evaluation unit 140 compares each word and assigns a weight to the utterance if the word is pronounced correctly or if the word is pronounced once again, and if a word is pronounced incorrectly or a word is omitted, it assigns a weight to the provided sentence. Calculate the matching rate.

Lastly, regarding the third method, the evaluation unit 140 classifies the provided sentences and utterances into word units. And as shown in FIG. 6, the evaluation unit 140 sets the number of cases for the classified word to a plurality of cases and then calculates the matching rate using the set number of cases.

Next, the misrecognition word collection unit 150 extracts unrecognized words or words that the learner fails to pronounce from the evaluated utterance and stores the extracted words in a table (S260).

The misrecognition word collection unit 150 has a high probability of not properly recognizing numbers that are difficult to recognize, words that are not used frequently, or words expressing human emotions and being converted incorrectly, so the misrecognition word collection unit 150 Extracts misrecognized words and stores the extracted words in table form.

The reason for storing misrecognized words in table form is to achieve an accurate recognition rate by comparing them with the utterance and analyzing them as incorrect or correct.

Finally, the control unit 160 outputs the analysis results for the utterance with a high matching rate (S270).

As shown in FIG. 7, the control unit 160 analyzes and outputs pronunciation accuracy, stress, and degree of similarity to intonation according to country.

At this time, as shown in FIG. 8, the control unit 160 acquires a waveform using the learner's uttered voice, and matches the obtained waveform with the time information of the word included in the utterance to obtain the stress of the word.

Meanwhile, the control unit 160 may determine the stress of a word according to the strength of the learner's voice.

In this way, according to the present invention, based on deep learning and natural language processing (NLP) technology, the user's level of speech for the sentence is evaluated, and the evaluated results are indexed and provided, so that the user can feel a sense of accomplishment and learn English on their own. By entering the pronunciation of , the location of the incorrect pronunciation is pointed out in units of phonemes or syllables and notified to the user. At the same time, detailed analysis of pronunciation accuracy, stress, intonation, speed, etc. can be performed, thereby improving learning efficiency.

In addition, according to the present invention, the recognition rate of the user's pronunciation can be increased by monitoring words that the user repeatedly pronounces incorrectly and continuously updating the conversion table, and classifying them into stressed words and unstressed words. By doing so, the matching rate between the uttered sentence and the provided sentence can be increased.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the patent claims below.

100: Pronunciation evaluation system
110: Text provision unit
120: Speech voice collection unit
130: Text conversion unit
140: Evaluation department
150: Misrecognition word collection unit
160: control unit

Claims (8)

In a pronunciation evaluation system based on deep learning and NLP (natural language processing) technology,
A text provider that provides sentences necessary for learning,
A spoken voice collection unit that collects the voice uttered by the learner in response to the provided sentence;
A text conversion unit that converts the uttered voice into text form and obtains the converted text as an utterance,
An evaluation unit that calculates a matching rate for sentences by comparing and analyzing the obtained utterance and the provided sentence using a previously constructed deep learning model;
A misrecognition word collection unit that uses the evaluated results to collect information on words that learners consistently pronounce incorrectly and words in which recognition errors occur, and stores the collected words in a table format;
If the utterance and the provided sentence match, the evaluation results are quantified and provided color-coded according to pronunciation accuracy and stress matching. If the utterance and the provided sentence do not match, a signal requesting reutterance for the provided sentence is sent. A pronunciation evaluation system including a control unit that According to paragraph 1,
The text conversion unit,
A pronunciation evaluation system that classifies the spoken voice by word, generates an utterance obtained using the classified words, and a plurality of candidate sentences similar to the utterance, and provides corresponding time information for each classified word. According to paragraph 1,
The evaluation department,
A pronunciation evaluation system that compares provided sentences and utterances word by word and calculates the matching rate based on the number of matched words. According to paragraph 3,
The evaluation department,
A pronunciation evaluation system that classifies provided sentences and utterances by word, compares each classified word, assigns weight to the provided sentences or utterances, and calculates the matching rate. According to clause 4,
The evaluation department,
If a word is pronounced correctly or a word is pronounced again, weight is given to the utterance,
A pronunciation evaluation system that gives weight to the sentences provided when words are mispronounced or words are omitted. According to clause 4,
The evaluation department,
A pronunciation evaluation system that classifies provided sentences and utterances by word, sets the number of classified words to a plurality of cases, and then calculates the matching rate using the set number of cases. According to paragraph 1,
The misrecognition word collection unit,
A pronunciation evaluation system that extracts numbers, infrequently used words, and words that learners repeatedly pronounce incorrectly, classifies them as misrecognized words, and stores the classified misrecognized words in a table. According to paragraph 1,
The control unit,
Obtain the waveform using the learner's spoken voice,
A pronunciation evaluation system that obtains the stress of a word by matching the obtained waveform with the time information of the word included in the utterance, compares the obtained stress with the stress of the original word, and outputs the correct answer in color.