KR20210147985A - Method and system for analyzing fluency using big data - Google Patents

Abstract

The present invention relates to a method for analyzing fluency using big data and a system thereof and, more specifically, to a method for automatically analyzing phenomena in which speech is not smooth, such as speech repetition, speech prolongation, blockage, speech correction, interjections, and the like, which are stuttering disorders, and a system thereof. To this end, a client comprises: a video/audio transmission unit transmitting video and audio to a server; a voice transmission unit transmitting voice to the server; and a result display unit displaying a fluency analysis result. The server comprises: a video/audio reception unit; the voice receiving unit; a voice analysis unit separating only the voice from a voice or video stream; a preprocessing unit removing noise from the voice; a minimum voice unit separation unit separating the voice into a minimum unit to perform detection; a voice feature extraction unit; a primary repeated and extended interjection prediction unit automatically predicting a non-fluency type on the basis of one-syllable repetition, extension, and interjection; a voice recognition unit; a language analysis unit performing morpheme analysis, syntax analysis, and semantic analysis; a secondary repeated and extended interjection prediction automatically predicting a non-fluency type on the basis of repetitive recognition of two or more syllable words, phrases, and clauses, extension of two or more syllables, and interjection; a correction blockage prediction unit automatically predicting a non-fluency type of blockage; and a result analysis unit calculating fluency analysis results.

Description

Method and system for analyzing fluency using big data}

The present invention relates to a method and system for analyzing fluency, and more particularly, if the flow of speech, such as repetition, prolongation, blockage, etc. of speech corresponding to a stuttering disorder, is not smooth It relates to a method and system for automatically analyzing an unexpected phenomenon.

Fluency Disorders are disorders in which communication is difficult due to an unnatural flow of speech.

Stuttering refers to abnormally frequent interruptions in the flow of speech, irregular speech speed, or unnecessary effort when speaking.

Rapid speech refers to a phenomenon in which speech gradually increases in speed during speech, resulting in poor articulation.

Currently, there is the Paradise-Fluency Test (P-FA) tool to analyze fluency, and based on this tool, a person can directly transcribe a voice to analyze normal non-fluency (ND) and abnormal non-fluency (AD).

Normal non-fluency (ND) includes hesitation (H), hepatic projection (I), incomplete or/and modified (Ur), repeat 1 (R1), and abnormal non-fluency (AD) hesitation-abnormal (Ha), hepatic-projection- Abnormal (Ia), incomplete or/and modified-abnormal (URa), Repetitive 1 - Abnormal (R1a), Repetition 2 (R2), and non-prosody phonation (DP).

However, this method is a method that takes a lot of time and cost because a person directly hears the sound and transcribes it into text to analyze the ND and AD.

As a prior art, a developmental disorder that searches databaseized speech therapy histories, extracts several treatment histories most similar to the current child's symptoms and utterances, and reprocesses the treatment histories to create and provide a new individual speech therapy curriculum. A child speech therapy device has already been developed (refer to KR Patent Publication 2019-0031128).

KR Patent Publication No. 2019-0031128 (2019.03.25)

The present invention has been proposed to solve the problems of the prior art, as described above, by collecting and learning voices for a section in which existing stuttering occurs, and automatically predicts stuttering when a new voice is input, thereby reducing human labor. To do this, extract the stuttering disorder voice and tag the stuttering disorder type corresponding to the section in which the voice occurred, and after feature extraction and learning for the stuttering disorder voice section, automatic extraction of the stuttering disorder section from the newly input voice and the type of stuttering disorder It aims to provide a method and system for analyzing fluency using big data that automatically predicts

The system for analyzing fluency using big data to achieve this purpose is composed of a client and a server.

Specifically, the client includes a video/audio transmitter that captures video and audio in real time through a camera and microphone and transmits it to the server as a stream, a voice transmitter that captures audio in real time through a microphone and transmits it to the server as a stream, and fluency analysis It is characterized in that the result display unit for displaying the result in a graph is configured.

In addition, the server separates only the audio from the video/audio receiver that receives the video and audio transmitted from the video/audio transmitter of the client in real time, the audio receiver that receives the audio transmitted from the audio transmitter in real time, and the audio or video stream. Voice analysis unit, pre-processing unit that removes noise from voice, voice minimum unit separation unit that separates and detects voice into minimum units, voice feature extraction unit that extracts features from voice, 1-syllable repetition, extension, and inter-projection based on voice The primary prediction unit between repeated extensions that automatically predicts the non-fluency type, the voice recognition unit that recognizes the voice, the language analysis unit that performs morpheme analysis, syntax analysis, and semantic analysis on the speech recognized text, speech recognition and language Recognition of repetition of words, phrases, and clauses of more than 2 syllables based on analysis, extension of more than 2 syllables, secondary prediction unit between repeated extensions that automatically predicts the non-fluency type of inter-projection, correction based on speech recognition and language analysis, non-fluency of blockage It is characterized in that the correction blockage prediction unit that automatically predicts the type, and the result analysis unit that calculates the fluency analysis result are configured.

And, in order to collect and learn the non-fluency type, and to generate a fluency evaluation model, a repeat-extension inter-projection input unit for inputting a voice to which repetition, extension, and inter-projection are applied; a learning voice feature extraction unit for extracting voice features to be used for learning; a machine learning learning unit that learns repeated, extended, and inter-projected voices through supervised learning; and a fluency evaluation model generator capable of evaluating fluency, and is characterized in that, when a voice is input, the non-fluency type of the corresponding voice is automatically predicted.

In addition, it automatically analyzes fluency even when video and audio are input at the same time or when only voice is input, 1 syllable repetition, 1 syllable extension, and 1 syllable projection automatically predict the non-fluency type based on the voice, words of 2 syllables or more, Phrase and clause repetition recognition, extension of more than two syllables, and inter-projection are characterized by automatically predicting non-fluency types based on speech recognition and language analysis.

On the other hand, the method of analyzing fluency using big data of the present invention includes the steps of capturing video and audio in real time through a camera and a microphone of a client and transmitting it as a stream to a server; Receiving the video and audio transmitted from the video/audio transmission unit in real time by the server; A method comprising: capturing voice through a client's microphone in real time and transmitting it as a stream to a server; A server real-time receiving the voice transmitted from the voice transmission unit; separating only the audio from the audio or video stream; removing noise from the voice; Separating and detecting a voice into a minimum unit; It is characterized in that the step of extracting the features from the voice is configured.

In addition, automatically predicting the non-fluency type based on the one-syllable repetition, extension, and inter-projection; Recognizing a voice; performing morphological analysis, syntax analysis, and semantic analysis on the speech-recognized text; automatically predicting non-fluency types such as word, phrase, and verse repetition recognition of more than two syllables, extension of more than two syllables, and inter-projection based on speech recognition and language analysis; automatically predicting non-fluency types such as correction and blockage based on speech recognition and language analysis; calculating a fluency analysis result, and displaying the fluency analysis result as a graph by the client; It is characterized in that it comprises a.

In addition, the method includes the steps of: collecting and learning non-fluency types, and inputting a voice to which repetition, extension, and inter-projection are applied to generate a fluency evaluation model; extracting features of speech to be used for learning; Learning repetition, extension, and inter-projection voice through supervised learning; and generating a fluency evaluation model capable of evaluating fluency, wherein when a voice is input, the non-fluency type of the corresponding voice is automatically predicted.

In addition, the method automatically analyzes fluency even when video and audio are input at the same time or only voice is input, and automatically predicting the non-fluency type based on voice for 1-syllable repetition, 1-syllable extension, and 1-syllable projection; And it is characterized in that it comprises a step of automatically predicting the non-fluency type based on speech recognition and language analysis for two or more syllables or more of word, phrase, and clause repetition recognition, extension of two syllables or more, and inter-projection.

The method and system for analyzing fluency using the big data of the present invention configured as described above exhibit the following useful effects.

1) It is possible to reduce the cost and time for fluency evaluation by automating the task of an expert listening to voice and transcribed it into text to evaluate fluency.

2) It automatically analyzes fluency even when video and audio are input at the same time or only audio is input.

3) It analyzes fluency by processing video and audio in real time without separate video or audio recording.

4) Fluency can be analyzed by collecting video and audio from various multimedia equipment such as robots, tablets, smartphones, electronic pens, holograms, digital signage, TVs, and automobiles in real time.

1 is a block diagram of a system for analyzing fluency using big data according to the present invention;
2 is a result display unit summary screen of a system for analyzing fluency using big data;
3 is a graph screen of the result display unit of a system for analyzing fluency using big data;

Hereinafter, preferred embodiments in which the object of the present invention can be specifically realized will be described in detail with reference to the accompanying drawings. In describing the present embodiment, the same names are used for the same components, and an additional description thereof will be omitted.

In the present invention, word repetition refers to repeating a syllable or a part of a word, a word, a phrase, a clause, etc. several times without stopping by itself. That is, it corresponds to the case where the syllable “(ah)” is repeated in “(ah) (ah) (ah) daddy”.

Prolongation of speech refers to a state in which speech sounds are continued for more than 0.5 seconds. In other words, it means that it is pronounced as "(ah~~~)" from "(ah~~~) daddy".

Speech blockage is a condition in which speech stops and it is no longer possible to continue the sound. ""Oh ah ah ah daddy (speak the next sentence without uttering for more than 2 seconds) Come".

The modification of the word refers to the utterance of “(Now Dinosaur)” in “(Now, Dinosaur), read a book, Daddy”.

Ganthuja refers to the utterance "(uh)" in "Dad (uh) went to (uh) work today."

1 is a block diagram of a system for analyzing fluency using big data according to the present invention.

As shown in FIG. 1, the system for analyzing fluency using big data according to the present invention is a client 100 and a server that analyzes fluency by processing video information input by a camera and voice information input into a microphone in real time (200).

The client 100 is composed of an audio transmission unit 101 , an image/audio transmission unit 102 , and a result display unit 103 .

The server 200 includes a voice receiver 201, an image/audio receiver 202, a voice analysis unit 203, a preprocessor 204, a minimum voice unit separation unit 205, a voice feature extraction unit 206, and repeat. Projection between extension primary prediction unit 207, voice recognition unit 208, language analysis unit 209, repeated projection secondary prediction unit 210, correction blockage prediction unit 211, result analysis unit 212, It consists of a projection voice input unit 213 between repeated extensions, a voice feature extraction unit 214 for learning, a machine learning learning unit 215 , and a fluency evaluation model generation unit 216 .

The video/audio transmission unit 102 of the client 100 performs a role of capturing video and audio in real time through the client-side camera and microphone and transmitting the video and audio as a stream to the server 200 .

The video/audio receiver 202 of the server 200 receives the video and audio transmitted from the video/audio transmitter in real time and transmits it to the audio analyzer.

The voice transmission unit 101 of the client 100 captures the voice in real time through the client-side microphone and transmits it to the server 200 as a stream.

The voice receiver 201 receives the voice transmitted from the voice transmitter 101 in real time.

The audio analysis unit 203 separates only the audio from the audio or video stream.

The preprocessor 204 removes noise from the voice.

The voice minimum unit separation unit 205 separates and detects the voice into the smallest unit.

The voice feature extraction unit 206 extracts features from the voice.

The repeat-extension inter-projection primary prediction unit 207 automatically predicts the non-fluency type based on one syllable repetition, extension, and inter-projection based on the voice.

The voice recognition unit 208 recognizes a voice.

The language analysis unit 209 performs morpheme analysis, syntax analysis, and semantic analysis on the speech-recognized text.

The repeated extension inter-projection secondary prediction unit 210 automatically predicts non-fluency types such as word, phrase, and clause repetition recognition of two or more syllables, extension of two syllables or more, and inter-projection based on speech recognition and language analysis.

The correction blockage prediction unit 211 automatically predicts the type of non-fluency such as correction and blockage based on speech recognition and language analysis.

The result analysis unit 212 calculates a fluency analysis result.

The result display unit 103 of the client 100 displays the fluency analysis result in a graph.

And, in the present invention, in order to collect and learn non-fluency types and to generate a fluency evaluation model, the repeated-extension inter-projection input unit 213 inputs a voice to which repetition, extension, and inter-projection are applied.

The learning voice feature extraction unit 214 extracts voice features to be used for learning.

The machine learning learning unit 215 learns repetition, extension, and inter-projection voice through supervised learning.

The fluency evaluation model generation unit 216 generates a model capable of evaluating fluency.

Through this, when a voice is input, the non-fluency type of the corresponding voice can be automatically predicted.

In addition, the system for analyzing fluency using big data of the present invention automatically analyzes fluency even when video and audio are simultaneously input or only audio is input.

And, 1-syllable repetition, 1-syllable extension, and inter-syllable projection automatically predict the non-fluency type based on speech, and the repetition recognition of words, phrases, and verses over 2 syllables, extension of 2 syllables or more, and inter-projection are based on speech recognition and language analysis. It is characterized by automatically predicting the non-fluency type.

In addition, the method of analyzing fluency using big data according to the present invention includes capturing video and audio in real time through a client-side camera and microphone and transmitting it to the server as a stream.

The video/audio transmitter receives the video and audio transmitted in real time.

Through the client-side microphone, the voice is captured in real time and transmitted to the server as a stream.

A step of receiving the voice transmitted from the voice transmitter in real time is performed.

It goes through a step of separating only the audio from the audio or video stream.

It goes through the steps of removing noise from the voice.

A step of separating and detecting a voice into a minimum unit is performed.

It goes through the steps of extracting features from the voice.

It automatically predicts the non-fluency type based on one syllable repetition, extension, and liver projection.

It goes through the steps of recognizing a voice.

It goes through the steps of morphological analysis, syntax analysis, and semantic analysis on the speech-recognized text.

Based on speech recognition and language analysis, it automatically predicts non-fluency types such as word, phrase, and clause repetition recognition of more than two syllables, extension of more than two syllables, and inter-projection.

Based on speech recognition and language analysis, it automatically predicts non-fluency types such as correction and blockage.

It goes through the steps of calculating the results of the fluency analysis.

It consists of displaying the results of the fluency analysis in a graph.

In addition, the present invention goes through the steps of inputting a voice to which repetition, extension, and inter-projection are applied in order to collect and learn non-fluency types and to generate a fluency evaluation model.

It goes through the steps of extracting the features of speech to be used for learning.

Through supervised learning, it goes through the steps of learning repetition, extension, and inter-projection speech.

It consists of creating a fluency evaluation model that can evaluate fluency.

When a voice is input through this, the non-fluency type of the corresponding voice is automatically predicted.

In addition, the present invention automatically analyzes fluency even when video and audio are input at the same time or only audio is input.

And, 1 syllable repetition, 1 syllable extension, and 1 syllable projection go through the steps of automatically predicting the non-fluency type based on the voice.

In addition, the repetition recognition of words, phrases, and clauses of more than two syllables, extension of more than two syllables, and inter-projection go through the steps of automatically predicting the non-fluency type based on speech recognition and language analysis.

division Contents Analysis method Analysis content repetition of speech (1 syllable) (Ah) (Ah) (Ah) Dad speech analysis extension of speech (1 syllable) (Ah~~~) Dad Liver Projection (1 syllable) Dad (uh) went to work today (uh) repetition of speech (more than 2 syllables) (here) (here) and Voice Recognition +
Language analysis (morpheme analysis, syntax analysis, semantic analysis) Extension of speech (more than 2 syllables) (Ji~~~Fri) Dad Liver projection (more than 2 syllables) Daddy (you know) a long-lived beetle is here. fertilization of horses (Now Dinosaur) Now Daddy reads a book The part excluded from the semantic relation (verb, verb, modifier, dialogue element) of the sentence blockage of words (Ah) (Ah) (Ah) Daddy (Speak the next sentence without uttering for more than 2 seconds) Come

As can be seen from Table 1, in the present invention, "repeating words" refers to repeating a syllable or a part of a word, a word, a phrase, a clause, etc. several times without stopping by itself.

"Repetition of words based on one syllable" corresponds to a case in which the "(ah)" syllable repeatedly occurs in "(a) (ah) (ah) daddy".

"Extension of speech based on one syllable" refers to a state in which speech sounds are continued for more than 0.5 seconds. In other words, it means that it is pronounced as "(ah~~~)" from "(ah~~~) daddy".

In the case of “one-syllable-based liver projection”, it corresponds to the case where the “(uh)” syllable occurs in “Dad (uh) went to (uh) today (uh) work”.

In the case of "one syllable-based speech repetition, speech extension, liver projection", after extracting features from the input speech, it is possible to predict whether "speech repetition", "speech extension" or "liver projection" is compared with the learning model.

When "repetition of speech", "prolongation of speech", and "liver projection" occur with more than two syllables, it can be inferred through speech recognition results and language analysis results, not speech analysis.

For example, in the case of "repetition of speech", since "here" is analyzed as a pronoun by morpheme analysis after voice recognition, it can be determined as repetition.

In the case of “now~~~~ now”, which is “extension of speech,” it can be determined as “extension of speech” because the combined word is analyzed as “now” and into a noun by morphological analysis after voice recognition.

In the case of "liver projection", "you are" is analyzed as a general verb "to be" by morphological analysis, and "you are" is registered in the liver projection dictionary, so it can be determined as "liver projection".

In the case of "modification of words", the relationship between the actor, the experiencer, the owner, the co-existent, the beneficiary, the object, the corpse, and the quotation/creation in the body part (the role of the subject and the object in the sentence) by syntax analysis and semantic analysis is sentenced In the verb part, the relationship between action, description, state statement, substantive statement, negative statement, verb expression, and verb expression is derived from the sentence.

In the modifier section (modifying adjectives, verbs, and modifiers in a sentence), the place, tool, negation, time, reason, condition, comparison, representation, and concession relations are derived from the sentence.

In addition, as dialogue elements, sentence relationships are derived by drawing attention, asking/confirming, exclamation, yes/no answer, emphasis, accompanying sound, greeting, connection, and automatic phrase.

The parts other than the verbal part, the verb part, the modifier part, and the dialogue element part are determined as the "word correction" candidate group.

Therefore, in Table 1, "correction of words" refers to the utterance of "(now dinosaur)" in "(now dinosaur), now Dad read a book".

That is, when (now) the actor (dad) acts (reads) the object (book). The preceding "now dinosaur" is an utterance that does not belong to an individual semantic type constituting the sentence.

Speech blockage is a condition in which speech stops and it is no longer possible to continue the sound. "Ah ah ah ah daddy (speak the next sentence without uttering for more than 2 seconds) Come"

2 is a summary screen of the result display unit of a system that analyzes fluency using big data.

On the summary screen of the result display unit of the present invention, language comprehension and language expression levels can be expressed in the form of developmental age.

In addition, it is possible to display the level of repetition of speech, the level of extension of speech, the level of blockage of speech, and the level of liver projection.

The index is the average incidence rate per 100 utterances.

3 is a graph screen of the result display unit of a system that analyzes fluency using big data.

On the graph screen of the result display unit of the present invention, graphs of the degree of non-fluency such as the repetition level of speech, the level of extension of the speech, the level of blockage of the speech, the liver projection, etc. are displayed as graphs by age, period, and level.

For example, the current treatment target is 6 years of age and 31 months of speech development, and the repetition level of speech is 50%, meaning that "word repetition" occurs on average about 50 times for every 100 utterances.

Compared with children of the same age, it can be seen that in the case of normal children, "repeated speech" occurs on average by 8% per 100 utterances and by 20% in the case of children with language development disorders.

Statistics by period show the progress of improving the current level of fluency by day, week, month, and year.

As such, the preferred embodiments according to the present invention have been reviewed, and the fact that the present invention can be embodied in other specific forms without departing from the spirit or scope of the present invention in addition to the above-described embodiments is common knowledge in the art. It is self-evident to those who have it.

Therefore, the above-described embodiments are to be regarded as illustrative rather than restrictive, and accordingly, the present invention is not limited to the above description but may be modified within the scope of the appended claims and their equivalents.

100...client 200...Server

Claims (6)

A video/audio transmitter that captures video and audio in real time through a camera and microphone and transmits it to the server as a stream;
A voice transmission unit that captures voice in real time through a microphone and transmits it to the server as a stream; and
A client comprising a result display unit for displaying the fluency analysis result in a graph; and
a video/audio receiver for receiving the video and audio transmitted from the video/audio transmitter of the client in real time;
A voice receiver that receives the voice transmitted from the voice transmitter in real time;
A voice analysis unit that separates only the voice from the voice or video stream;
A preprocessor that removes noise from voice,
A voice minimum unit separation unit that separates and detects the voice into the smallest unit;
A voice feature extraction unit for extracting features from the voice,
A primary prediction unit between repeated and extended projections that automatically predicts the non-fluency type based on speech-based 1-syllable repetition, extension, and inter-projection;
A voice recognition unit for recognizing a voice,
A language analysis unit that performs morphological analysis, syntax analysis, and semantic analysis on the speech-recognized text;
Based on speech recognition and language analysis, repeated recognition of words, phrases, and clauses of more than two syllables, extension of more than two syllables, and a secondary prediction unit between repeated extensions that automatically predict the non-fluency type of the inter-projection;
Correction blockage prediction unit that automatically predicts the non-fluency type of correction and blockage based on speech recognition and language analysis;
a server comprising a result analysis unit for calculating a fluency analysis result;
A system for analyzing fluency using big data, characterized in that it comprises a. The method of claim 1,
To collect and learn non-fluency types and create a fluency evaluation model,
a repeated and extended inter-projection input unit for inputting a voice to which repetition, extension, and inter-projection are applied;
a learning voice feature extraction unit for extracting voice features to be used for learning;
a machine learning learning unit that learns repeated, extended, and inter-projected voices through supervised learning; and
It consists of a fluency evaluation model generation unit that can evaluate fluency,
A system for analyzing fluency using big data, characterized in that when a voice is input, the non-fluency type of the corresponding voice is automatically predicted. 3. The method according to claim 1 or 2,
It automatically analyzes fluency even when video and audio are input at the same time or only audio is input.
1 syllable repetition, 1 syllable extension, and 1 syllable projection automatically predict the non-fluency type based on the voice,
A system that analyzes fluency using big data, characterized by automatically predicting the type of non-fluency based on speech recognition and language analysis for word, phrase, and clause repetition recognition of more than 2 syllables, extension of more than 2 syllables, and inter-projection. capturing video and audio in real time through the client's camera and microphone and transmitting it as a stream to a server;
Receiving the video and audio transmitted from the video/audio transmission unit in real time by the server;
A method comprising: capturing voice through a client's microphone in real time and transmitting it as a stream to a server;
A server real-time receiving the voice transmitted from the voice transmission unit;
separating only the audio from the audio or video stream;
removing noise from the voice;
Separating and detecting speech into minimum units
extracting features from speech;
automatically predicting a non-fluency type based on one-syllable repetition, extension, and inter-projection;
Recognizing a voice;
performing morphological analysis, syntax analysis, and semantic analysis on the speech-recognized text;
automatically predicting non-fluency types such as repeated recognition of words, phrases, and clauses of two or more syllables, extension of two or more syllables, and inter-projection based on speech recognition and language analysis;
automatically predicting non-fluency types such as correction and blockage based on speech recognition and language analysis;
calculating a fluency analysis result; and
displaying the fluency analysis result as a graph by the client;
A method of analyzing fluency using big data, characterized in that it comprises a. 5. The method of claim 4,
inputting a voice to which repetition, extension, and inter-projection are applied in order to collect and learn non-fluency types and generate a fluency evaluation model;
extracting features of speech to be used for learning;
Learning repetition, extension, and inter-projection voice through supervised learning; and
It consists of generating a fluency evaluation model that can evaluate fluency,
A method of analyzing fluency using big data, characterized in that when a voice is input, the non-fluency type of the corresponding voice is automatically predicted. 6. The method according to claim 4 or 5,
It automatically analyzes fluency even when video and audio are input at the same time or only audio is input.
1 syllable repetition, 1 syllable extension, and 1 syllable projection automatically predicting the non-fluency type based on the voice; and
Repeat recognition of words, phrases, and verses more than two syllables, extension of more than two syllables, and inter-projection are constructed including the steps of automatically predicting non-fluency types based on speech recognition and language analysis. How to analyze.

Images