KR20190041769A - Apparatus and method for evaluating linguistic performance based on pitch and interval of silence - Google Patents

Abstract

The present invention relates to an apparatus and method for evaluating linguistic performance based on a silence interval. The method may comprise: a user silence interval step that a user silence interval extracting module generates or receives a user silence interval, which is a silence interval of user voice data, based on user voice data; and a silence interval similarity calculation step that a silence interval similarity calculation module receives information for the user silence interval and calculates silence interval similarity, which is similarity between a reference silence interval, which is a silence interval of reference voice data, and the user silence interval. Accordingly, the apparatus and method may evaluate performance of language shadowing in real time (for example, within 100ms) and provide feedback.

Description

[0001] The present invention relates to an apparatus and method for evaluating linguistic performance based on silence intervals for outputting a main note,

The present invention relates to an apparatus and method for evaluating a linguistic performance based on a silence interval for outputting a main attention word.

Language education services are evolving into a new paradigm, driven by the high share of mobile devices and the enhanced computing power of devices. If the existing language education service was one-sided education contents delivery, recent language education service is a paradigm of interactive education with real-time feedback due to the development of technologies such as STT (Speech to text), NLP (Natural Language Processing) and webRTC . As a part of this interactive education, we are now improving the users' linguistic service experience by performing automatic evaluation of speaking in various language services such as Speak, Brain, and Chinatan.

As a method for evaluating user's speaking ability in the above-mentioned language education service, there is a method of judging whether a specific sentence and a user's input voice are similar using frequency analysis, a method of converting to text using STT (Speech to text) And how to determine whether the input voice of the user is similar to a specific sentence. These methods have been very helpful in elaborately assessing the user's speaking skills in the web or app.

Among the many existing language education services, there is "talking along" as one of brain training scientific meaningful learning methods. The follow-up method, which is a method for quickly forming an implicit memory of a specific language, is a method of simultaneously speaking while listening to a reference voice (usually a foreigner) for a specific sentence.

However, the existing language speech evaluation method (language performance evaluation) has a problem that the evaluation result of the language performance can not be obtained after the input voice of the user is terminated because the computing resource is too large, .

Speech evaluation system using mobile terminal and evaluation method thereof, Korean Patent Laid-open Publication No. 10-2016-0123872, Paichai University Industry-Academic Cooperation Foundation Speaking ability notarization system and method thereof, registered patent 10-0943477, Korean Broadcasting Corporation

Therefore, it is an object of the present invention to provide a speech recognition apparatus and a speech recognition method, which are capable of evaluating the performance of speaking based on language in real time, And to provide a device and a method for evaluating a linguistic performance based on a silent section.

Hereinafter, specific means for achieving the object of the present invention will be described.

An object of the present invention is to provide an activation step of allowing an activation module to sense a signal at the start of utterance of reference speech data in user speech data to start performing language performance evaluation; A user silence interval extraction module extracting a user silence interval that is a silence interval of the user voice data based on the user voice data and generating user silence interval information that is information on the user silence interval; And a silence interval similarity calculation module that receives the user silence interval information and calculates silence interval similarity, which is the similarity between the reference silence interval information, which is information on the silence interval of the reference speech data, and the user silence interval information, And evaluating the language performance of the user voice data based on the silence interval similarity. The method of evaluating the silence interval based language performance using the activation module may be provided.

Another object of the present invention is to provide a speech recognition apparatus and a speech recognition method, which can be realized by an activation module for detecting a signal at the start of utterance of reference speech data in user speech data and starting performance of language performance evaluation. A user silence interval extracting module for extracting a user silence interval that is a silence interval of the user voice data based on the user voice data and generating user silence interval information about the user silence interval; And a silence interval similarity calculation module that receives the user silence interval information and calculates silence interval similarity, which is the similarity between the reference silence interval information and the silence interval information, which is information on the silence interval of the reference voice data, And the linguistic performance of the user voice data is evaluated on the basis of the similarity degree of the silence section. The present invention can be achieved by providing a silence interval-based language performance evaluation apparatus using the activation module.

As described above, the present invention has the following effects.

First, according to one embodiment of the present invention, an effect that a high computing power is not required for language performance evaluation occurs.

Second, according to one embodiment of the present invention, it is possible to evaluate the performance of language-following speech in real time (for example, within 100 ms) and to feedback it.

Third, according to an embodiment of the present invention, language performance can be evaluated regardless of the user's voice recording distance or the level of the recording quality.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and, together with the description, And shall not be interpreted.
BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram showing a linguistic performance evaluation apparatus based on a silence period according to an embodiment of the present invention;
2 shows an example of a user's speech waveform,
FIG. 3 is a flowchart illustrating a silent interval-based language performance evaluation method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following detailed description of the operation principle of the preferred embodiment of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may unnecessarily obscure the subject matter of the present invention.

The same reference numerals are used for portions having similar functions and functions throughout the drawings. In the specification, when a specific portion is connected to another portion, it includes not only a direct connection but also a case where the other portion is indirectly connected with another element in between. In addition, the inclusion of a specific constituent element does not exclude other constituent elements unless specifically stated otherwise, but may include other constituent elements.

A linguistic performance evaluation device based on silence interval

1 is a schematic diagram showing a silence interval-based language performance evaluation apparatus according to an embodiment of the present invention. 1, the silence interval-based language performance evaluation apparatus 1 according to the embodiment of the present invention includes a user silence interval extraction module 10, a silence interval similarity calculation module 11, a language performance evaluation Module 20, and a reference silence interval extraction module 30. The user inputs the user voice data 100 to the linguistic performance evaluation device 1. The user voice data 100 is input in such a manner that the user client outputs the reference voice and speaks along with the reference voice, The user voice data 100 may be input in such a manner that the voice and the corresponding reference language content (e.g., text) are output and talked with it.

The user silence interval extraction module 10 is a module for generating information on a user silence interval which is a silence period of the user's voice data based on the user's voice data 100. The user silence interval extraction module 10 according to an embodiment of the present invention is provided in a user client and transmits information on a user silence interval to a server or generates information on a user silence interval directly in a server Lt; / RTI > When the user silence interval extraction module 10 is configured in the client and the silence interval similarity calculation module 11 is configured in the server, the user voice data 100 itself (PCM raw data) Only the silence section information is transmitted to the server without transmitting the data, so that the transmission / reception data can be lightened. When the user silent interval extraction module 10 according to the embodiment of the present invention is provided in the server, the user voice data 100 received from the user client may be transmitted as is or may be properly encoded and streamed to the server.

The user silence section extraction module 10 extracts a section of a user's voice data 100 having a specific dB or less (or a specific amplitude or less) from a voice waveform of the user's voice data 100 as a user's silence section extraction method of the user silence section extraction module 10, It can be extracted as silence section. 2 shows an example of a user's voice waveform. As shown in FIG. 2, a user silence interval extraction module 10 defines a section of a certain dB or less as a user silence interval 2, And the like. The information (user silence period information) about the user silence period according to the embodiment of the present invention includes the length (a) of the silent section, the temporal position (b) at the start of utterance, the number of silence periods from the start of utterance, The total length of the silent section, the time after the end of the previous silent section, and the like. By constructing the information on the silent section to be extracted as described above, a quick and precise evaluation of the language performance becomes possible.

The user silence interval extraction module 10 and the reference silence interval extraction module 30 according to an embodiment of the present invention may further include a silence interval extraction module 30 for calculating the silence interval similarity of the silence interval similarity calculation module 11, And the reference silence section information may further include at least one of signal information immediately before the silence section and signal information immediately after the silence section. Such signal information may include, for example, phoneme information or frequency information. If at least one of the silence interval information immediately before and after the silence interval is further included in the silence interval information extracted from the user silence interval extraction module 10 and the reference silence interval extraction module 30 according to an embodiment of the present invention, The matching between the silent section and the reference silent section is elaborated, and the determination of the similarity degree of the silent section is elaborated, so that the accuracy of the language performance evaluation is improved and the effect of enabling the observer to avoid only the silent section sink can be avoided.

The silence period similarity calculation module 11 receives information on the user silence period in the user silence period extraction module 10, receives information on the reference silence period in the reference silence period extraction module 30, Which is a similarity between the user silence interval and the user silence interval. The silence period similarity calculation module 11 according to an exemplary embodiment of the present invention is provided in each application of a plurality of users' clients to configure each client to calculate the silence interval similarity, which is the similarity between the reference silence period and the user silence period . In addition, the silence period similarity calculation module 11 according to an exemplary embodiment of the present invention may be provided in the server, and the server may calculate the silence interval similarity, determine the feedback according to the calculated silence interval similarity, and output the result to each client.

The silence period similarity calculation module 11 according to the embodiment of the present invention calculates the silence period similarity calculation method of the silence period similarity calculation module 11 based on the temporal position at the start of the speech included in the user silence period information, A reference silence period corresponding to a user silence period for which the similarity degree is to be calculated based on whether the reference silence period is from the start to the reference silence period and based on the user silence period information and the reference silence period information corresponding to the reference silence period, Can be calculated.

Particularly, in order to improve the accuracy of the similarity calculation, the time gap (Gap) between the reference speech data 300 and the user speech data 100 is corrected by aligning the time based on the end of the previous silence interval .

The silence period similarity calculation module 11 according to an exemplary embodiment of the present invention detects a specific number of user silence periods in the user's voice data 100 with respect to the silence period similarity calculation frequency of the silence period similarity calculation module 11, The silent interval similarity between the user silence interval and the corresponding reference silence interval may be calculated. According to this, evaluation of language performance can be performed within 100 ms to 500 ms, and real-time feedback on speaking based on language is enabled.

The language performance evaluation module 20 is a module for evaluating the language performance of the user speech data based on the silence interval similarity received by the silence interval similarity calculation module 11 to generate the language performance data 200. The result of the language performance evaluation for the user of the language performance evaluation module 20 can be used for determining the degree of difficulty of the next language content and the difficulty level of the next language content can be determined based on the speed of the reference speech data 300, Element. ≪ / RTI >

The language performance evaluation module 20 according to the embodiment of the present invention compares the silence section similarity degree information generated by the silence section degree similarity degree calculation module 11 with the user silence section The similarity of the reference silence interval information with the reference silence interval information can be evaluated to be successfully performed according to the language of the reference speech data 300.

Alternatively, the linguistic performance evaluation module 20 may reflect the interval length weight, which is a weight for at least one of the user silence interval and the reference silence interval, in the silence interval similarity calculation module 11 so that the longer the length of the silence interval The degree of similarity of the silent section may be configured to have a greater effect on the evaluation result of the language performance. When the language performance evaluation module 20 is configured as described above, there is an effect that it is not necessary to consider which silence section should or should not be reflected in the evaluation algorithm in the reference speech. Therefore, there is an effect that it is possible to perform an evaluation which is hardly affected by a change in the silent section pattern due to the speech speed of the reference voice. If we select the silent section to be evaluated with a fixed length, there is a problem that performance difference occurs in the voice of a person having a fast speech and the voice of a person having a slow speech.

The reference silence section extraction module 30 is a module for generating information about a reference silence section (reference silence section information), which is a silence section of the reference data, based on the reference speech data 300. The reference silence interval extraction module 30 according to an embodiment of the present invention is provided in a user client and receives information on the reference speech data 300 from a server or is provided in a server and is provided with information about a reference silence interval . ≪ / RTI >

The reference silent section extraction module 30 refers to the reference silent section extraction method of the reference silent section extraction module 30 by referring to the section of the speech sound waveform of the reference speech data 300 It can be extracted as silence section. The information about the reference silence section (reference silence section information) according to the embodiment of the present invention includes the length of the silence section, the temporal position at the start of the utterance, the number of silence sections from the start of utterance, the total length of the silence section , The time after the end of the previous silence period, and the like. The reference silence interval extraction module 30 according to an embodiment of the present invention may extract the reference silence interval information in real time, and extract the reference silence interval information in advance and store the information in the server or the client.

When the language performance is evaluated through comparison between the user voice data 100 and the reference voice data 300, the silence interval information (the length of the silence section, the temporal position at the start of the utterance, the number of silence periods from the start of utterance, It is possible to evaluate the user's language performance very quickly (within 100 ms to 500 ms) when the similarity of the silence interval from the start of speech to the silence interval, the time after the end of the previous silence interval, the signal information immediately before the silence interval, And there is an effect that can be applied to situations where fast difficulty feedback is required, such as speaking language.

Evaluation method of linguistic performance based on silence interval

FIG. 3 is a flowchart illustrating a silent interval-based language performance evaluation method according to an embodiment of the present invention. As shown in FIG. 3, the silent interval-based language performance evaluation method according to an embodiment of the present invention includes a user silence interval extraction step S10, a similarity degree determination step S20, and a language performance evaluation step S30 .

In the user silence interval extraction step S10, the user silence interval extraction module 10 extracts a user silence interval, which is a silence interval of the user's voice data, from the user's voice data 100, and a user silence interval Information generating step. The information on the user silence section (user silence section information) according to the embodiment of the present invention includes the length of the silence section, the temporal position at the start of the utterance, the number of silence periods from the start of utterance, , The time after the end of the previous silence period, and the like.

In the similarity determination step S20, the silence period similarity calculation module 11 receives the information on the user silence period from the user silence period extraction module 10, and the reference silence period extraction module 30 extracts the reference silence period And calculates the silence interval similarity, which is the similarity between the reference silence period and the user silence period.

The language performance evaluation step S30 is a step in which the language performance evaluation module 20 evaluates the language performance of the user speech data based on the silence interval similarity received by the silence period similarity degree calculation module 11. [

Variation example

[ First Modification - The signal at the firing point. Activation As a trigger ]

According to the first modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to the embodiment of the present invention may further include an activation module. The activation module according to the first modification of the present invention can be configured to detect only the signal at the start of speech of the reference speech data 300 in the user speech data 100 and to start performing the language performance evaluation. In the case where the activation performance module is further included according to the first modification of the present invention, when the language performance evaluating apparatus 1 is configured in the client, the power consumption is reduced, and the silence period similarity degree calculating module 11 is configured in the server The effect of reducing the communication use amount is generated.

[ Second Modification - Difficulty tagging time to get to a certain level]

According to the second modification of the present invention, the difficulty tagging module may be further included in the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention. The difficulty tagging module according to the second modification of the present invention tracks how many times the user must perform repetition to converge to a result of a certain degree of similarity or more when the user performs linguistic speaking on a specific reference speech, The linguistic difficulty of the voice can be determined and tagged. For example, if the user converges to a result of more than a certain degree of similarity by saying two times, if the difficulty tagging module tags the reference voice with 'difficulty low', the user converges to a result more than a certain degree of similarity The difficulty tagging module can be tagged with the reference voice as 'difficulty high'. According to this, there is an effect that the difficulty tagging can be automatically performed for a plurality of reference voices according to the performance of speech by the user.

[ Third Modification - Determine next language content]

According to the third modified example of the present invention, the linguistic performance output apparatus based on the silence interval-based language performance according to the embodiment of the present invention may further include a language content output module. The language content output module according to the third modified example of the present invention tracks when the user performs speech according to the language of a specific reference speech to see how many times the user has to repeat to converge to a result of more than a certain degree of similarity, And determines the language content to be output to the degree of difficulty according to the degree of difficulty. For example, when the user converges to a result of more than a certain degree of similarity by saying two times, the language content output module outputs linguistic content having a higher level of difficulty than the reference voice. When the user speaks 10 times, The linguistic content output module outputs linguistic content having a lower degree of difficulty than the reference voice. According to this, the effect of outputting the linguistic content in which the degree of difficulty is automatically adjusted according to the linguistic follow-up performance of the user is generated.

[ Fourth Modification - In the silence section, the sound height information is further compared]

According to the fourth modification of the present invention, the silence interval information analyzed by the silence interval-based language performance evaluating apparatus 1 according to the embodiment of the present invention further includes the pitch height information (Pitch) between the silence intervals . According to the fourth modification of the present invention, when the silence interval information further includes the sound level information between the silence periods, an effect of enabling a more elaborate language performance evaluation is obtained.

[ Fifth Modification - Similarity to previous order The degree of change  It is judged that the specific level is reached when the temperature is low]

According to the fifth modified example of the present invention, in the language performance evaluation module 20 of the silence interval-based language performance evaluation apparatus 1 according to the embodiment of the present invention, If the change in the degree of similarity is less than or equal to a certain value, it can be regarded as success for the reference speech and the next language content can be output. According to the fifth modification of the present invention, when the information about the change in the degree of similarity is added to perform the language performance evaluation, there is an effect that the user-customized language performance evaluation becomes possible.

[ Sixth Modification - When a certain level is reached, tag the phrase for the pronunciation of a specific phrase.

According to a sixth modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention may further include an pronunciation tagging module, The tagging module can be configured to tag the corresponding phrase for user voice data at a certain degree of similarity or higher. According to the sixth variation of the present invention, when the pronunciation tagging module tags the corresponding phrase for the user voice data, the pronunciation is tagged automatically for the phrase according to the user's linguistic follow-up performance.

[ Seventh Modification - Only in previous phrases of different silence periods STT  compare]

According to the seventh modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention may further include a main attention word output module. According to the seventh variant of the seventh variant of the present invention, the text data of the reference language content and the text data of the user voice data are converted into STT (Speech to text) for the previous phrase of the silence section in which the similarity degree of the silence section is low , It is possible to output information indicating which part of the reference language content should be more carefully cared for by the language. According to the seventh modification of the present invention, when the main attention word output module is used, an effect of giving improved feedback to the user with a small calculation amount is generated.

[ Eighth Modified Example - Adjustment of difficulty by output speed of reference voice data]

According to the eighth modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention may further include a difficulty level adjusting module. The difficulty level adjustment module of the eighth modification of the present invention adjusts the output speed of the reference voice, in particular, the reference voice, which is the language content to be outputted next, according to the language performance of the user evaluated by the language performance evaluation module Lt; / RTI > For example, when the language performance is evaluated as 'high', if the output speed of the reference voice outputted this time is 100 words / minute, the output speed of the reference voice to be outputted next is adjusted to 150 words / minute. According to the eighth modification of the present invention, when the output speed of the reference voice to be output next is adjusted according to the language performance of the user, the user experiences an appropriate degree of difficulty.

[ Ninth Modification - the user provides a reference voice]

According to the ninth modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention may further include a reference speech receiving module. The reference voice receiving module of the ninth modification of the present invention can receive the reference voice to the user and convert it into a reference language content using the STT and then provide the reference voice and the reference language content to other users. According to this, the effect of increasing the amount of the reference voice data in accordance with the user's activity occurs.

[ Tenth modification - users provide reference language content]

According to a tenth modification of the present invention, the silence interval-based language performance evaluating apparatus 1 according to an embodiment of the present invention may further include a reference language content receiving module. The reference language content receiving module of the tenth modification of the present invention receives a text-based reference language content from a user and converts it into reference voice data using TTS (Text to Speech), and provides the reference speech and the reference language content to other users can do. According to this, the effect of increasing the amount of the reference language content depends on the activity of the user.

[ Eleventh Modification - Using other users' silence interval similarity in language performance evaluation]

According to the eleventh modification of the present invention, the language performance evaluation module 20 of the silence interval-based language performance evaluating apparatus 1 according to the embodiment of the present invention calculates the silence interval When performing the linguistic performance evaluation of the user, the silence interval similarity of other users can be used. According to this, the effect of statistically evaluating users' language achievement can be obtained without setting a separate threshold.

[ Twelfth Modification Example - When extracting silence section Fury  Conversion use]

According to a twelfth modified example of the present invention, the user silence interval extraction module 10 and the reference silence interval extraction module 30 of the silence interval-based language performance evaluation apparatus 1 according to the embodiment of the present invention are configured to extract, The frequency is separated by the Fourier transform when extracting the silence period, the frequencies with the small changes in dB among the separated frequencies are regarded as background sounds, and the interval in which only the background sound is remaining in the voice data is extracted as the silence period. According to this, there is an effect that a uniform language performance can be evaluated despite the distance between the user and the microphone, the background noise, and the like.

[ Modification 13 - Time to success. With a reward  Personalized by reinforcement learning]

As described above, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention.

The features and advantages described herein are not all inclusive, and in particular, many additional features and advantages will be apparent to those skilled in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used herein is primarily chosen for readability and for purposes of teaching, and may not be selected to delineate or limit the subject matter of the invention.

The foregoing description of embodiments of the invention has been presented for purposes of illustration. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will appreciate that many modifications and variations are possible in light of the above teachings.

The scope of the invention is, therefore, not to be limited by the Detailed Description, but is to be defined by the claims of any application based thereon. Accordingly, the disclosure of embodiments of the invention is illustrative and not restrictive of the scope of the invention, which is set forth in the following claims.

1: Language performance evaluation device based on silence interval
2: Silent section
10: user silence interval extraction module
11: Silence interval similarity calculation module
20: Language Performance Evaluation Module
30: Reference silent section extraction module
100: User voice data
200: Language performance data
300: reference voice data
a: length of the silence section
b: temporal position at the start of speech

Claims (2)

A user silence interval extraction module extracting a user silence interval that is a silence interval of the user voice data based on user voice data and generating user silence interval information that is information on the user silence interval;
A silence interval similarity calculation step of calculating a silence interval similarity between the reference silence interval information and the user silence interval information, the silence interval similarity calculating module receiving the user silence interval information and the silence interval information of the reference speech data;
A language performance evaluation step in which the language performance evaluation module evaluates the language performance of the user voice data based on the silence interval similarity; And
The main attention word output module compares the text data of the reference speech data with the text data of the user speech data by using STT (Speech to text) for the previous phrase of the silence section in which the similarity degree of the silence section is low, Outputting the main attention word outputting information of the main attention word, which is information on which part of the data should be more specifically cautiously followed by speech-based speech;
/ RTI >
A method of evaluating linguistic performance based on silence interval for output of major notes.
A user silence interval extracting module for extracting a user silence interval that is a silence interval of the user voice data based on the user voice data and generating user silence interval information about the user silence interval;
A silence interval similarity calculation module that receives the user silence interval information and calculates a silence interval similarity that is a similarity between the reference silence interval information and the silence interval information of the reference speech data;
A linguistic performance evaluation module for evaluating the linguistic performance of the user speech data based on the silence interval similarity; And
The speech data of the reference speech data and the text data of the user speech data are compared with STT (Speech to Text) for the previous phrase of the silence section in which the similarity degree of the silence section is low, A key attention word output module that outputs information of the main state of the word, which is information on whether to speak the language according to a more careful attention;
/ RTI >
A linguistic performance evaluation device based on the silence interval for the output of the main note.

Images