KR20200018859A - Web service system for speech feedback - Google Patents

Abstract

The purpose of the present invention is to detect and extract speech habits of a user, namely filler sound, to provide a web service for providing speech feedback. According to the present invention, to this end, a web system for speech feedback includes: a server communicating with a Google API server to receive information of a sound file which is converted into text, simultaneously running a Python library to analyze sound data, and transferring the analyzed sound data to a client device; a display displaying a web page configured for a user to upload the sound file and check information of the analyzed sound file; and a client transferring the sound file to the server and transferring the sound file received from the server to the display to display the sound file on the web page.

Description

Web service system for speech feedback {WEB SERVICE SYSTEM FOR SPEECH FEEDBACK}

The present invention relates to a web service for speech feedback. The present invention relates to a web service for improving a user's speech ability by analyzing and feeding back a user's speech.

The importance of speech has recently emerged. Speech is one of the abilities that everyone needs, especially one of the qualities that will help them perform their jobs.However, speech skills cannot be expanded in a short period of time, so constant practice is required. There is a problem.

Most of the speech schools in the market are coaching using the recorded video after recording, the schools are limited to large cities, and the monthly fee is a significant problem. In addition, although there is a mobile application that analyzes speech as shown in FIG. 1, this is only a level of feedback of instantaneous sound volume or speed of speech, and does not provide more useful information. There is no function to analyze the data, but it is just enough to make recording or practice more convenient. That is, a service such as an institute can provide accurate and specific feedback, but it consumes a lot of time and money, and in the case of an application, the feedback can be easily provided.

As shown in the results of the survey shown in FIGS. 2 to 5, there are a large number of users who need to correct the speech, and most users want to correct the speech in a more effective and convenient way. Most users have never encountered services, and none of them easily get help for speech correction.

An object of the present invention is to provide a web service system for providing speech feedback by detecting and extracting a speaking habit of a user, that is, a filler sound.

In order to achieve the above problem, the web service system for speech feedback of the present invention communicates with a Google API server to receive voice file information converted to text, and simultaneously executes a Python library to analyze voice data and analyze the analyzed voice. A server for delivering data to the client device; A display for displaying a web page configured to allow a user to check uploaded and analyzed voice file information of the voice file; And a client that delivers the voice file to the server or delivers the voice file received from the server to the display to display on the web page.

In particular, the Python library is characterized by including librosa, and the analysis of the speech data is characterized by including frequency-related feature value extraction and amplitude-related feature value extraction. In addition, the frequency-related feature value extraction is characterized by using the short time Fourier Transform (STFT) of librosa, the amplitude-related feature value extraction is characterized by using the RMSE of librosa.

The web page may include a start analysis button, a script, a legend button, a recommended video, and the like. The script may include a result of converting a voice file into text. It is characterized by a button for displaying a sudden change, a large decibel period, a fast section, etc. in the script.

The present invention has an effect that can provide speech practice without restriction on the place and time by detecting and extracting the speaking habits of the user, that is, the filler sound to provide speech feedback.

1 is a diagram illustrating a mobile application that analyzes conventional speech.
2-5 are graphs displaying the results of a survey indicating the need for speech correction.
6 is a diagram illustrating a result of converting a voice file into a script using the Google API.
7 to 10 are diagrams showing the results of using a short time fourier transform (STFT) for frequency-related feature value extraction among tools provided by librosa.
11 and 12 illustrate the results of using RMSE in amplitude-related feature value extraction among tools provided by librosa.
13 is a diagram showing a web system structure of the present invention.
14 is a view showing a web page according to an embodiment of the present invention.

The foregoing objects, features, and advantages will become more apparent from the following examples taken in conjunction with the accompanying drawings.

Specific structural to functional descriptions are merely illustrated for the purpose of describing embodiments in accordance with the inventive concept, and embodiments in accordance with the inventive concept may be embodied in various forms and are described in the specification of the present application. It should not be construed as limited to these.

Embodiments in accordance with the concepts of the present invention can be variously modified and have a variety of forms specific embodiments will be illustrated in the drawings and described in detail in the specification of the present application. However, this is not intended to limit the embodiments in accordance with the concept of the present invention to a specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof that is described, and that one or more other features or numbers, It is to be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements.

For user feedback, it is important to analyze important information such as 'filler sound' in the user's voice data. Filler sound is a meaningless but habitual spitting voice, for example, in Korean, 'sound', 'er', 'he', 'fact' and the like.

Machine learning can be used to develop a system for detecting filler sound. Machine learning includes a method using AWS and a model (eg, an original machine learning model) for classifying musical instruments to classify whether a given voice data is filler sound or not. Models using speech features extracted with librosa using AWS may have the disadvantage of inaccurate test results.

On the other hand, in order to show what characteristics the user has shown at what point, it is necessary to have a reference text. The research on converting speech into text is actively studied in the field of Automatic Speech Recognition (ASR). To convert speech into text, you also need to understand speech, but you also need to understand the language because you need to change it to the language that people actually use. In addition, the corpus data that people actually use is necessary, and since the data is huge, it is generally difficult to convert text unless it is a large corporation or research institute. Therefore, in the present invention, Google speech API which is a text conversion service provided by Google can be applied.

When the user uploads the voice data to Google Cloud, this file is analyzed and the information converted into text is transmitted in the form of a JSON file. The JSON file shows the text information of the voice and when each word starts and ends, as shown in FIG. Can be. Based on this, you can analyze the speed of scripts and words.

In addition, in the present invention, a python open source librosa may be applied to extract various feature values of speech. In general, voice files used in electronic devices mainly have a sampling rate of 44100 Hz, and librosa reads these digital signals again at a sampling rate of 22050 Hz. In general, the frequency range of a meaningful human voice is 0-3000 Hz, so librosa can be used to capture the characteristics of a human voice sufficiently because the sampling rate of librosa is much higher than the human voice frequency.

7 to 10 are diagrams showing the results of using a short time fourier transform (STFT) for frequency-related feature value extraction among tools provided by librosa.

STFT combines voice data into specific size units and performs Fourier transform for each unit. When the Fourier transform is calculated by combining 512 pieces of speech data read at a sampling rate of 22050 Hz, the Fourier transform is performed every 23 ms. Since 23ms is a short enough time to grasp the frequency change of the voice, the STFT can be used to calculate the frequency component change over time of the voice.

In FIG. 7, as a result of analyzing a recording file in which a user pronounces 'Doremipasolasi' one by one, STFT, X-axis represents time, Y-axis frequency, and brightness represents signal strength. Thus, three-dimensional data acquisition of speech is possible. As shown in the spectrum of FIG. 7, the sections with bright bands and the sections with no bright bands represent sections with and without speech, respectively. In addition, as the frequency band of the voice increases, the bright band region also increases in the high frequency region (positive Y-axis direction).

In the spectrum of FIG. 7, it is possible to check the presence or absence of voice (voice) and the intensity of each frequency component of the voice with respect to time. In the present invention, the data obtained from the spectrum of FIG. 7 may be used for feature value extraction, which may indicate a feature of the presentation required by the user. The frequency range of a significant human voice is on the order of 0-3000 Hz, but it is not the only dominant frequency for a particular time. As shown in the spectrum of FIG. 7, the human voice may be represented by a combination of several frequencies. Another feature of human voices is the fundamental frequency, which is the smallest value of the frequency components of a specific time, and the remaining frequencies may be expressed as integer multiples of the fundamental frequency.

Referring to FIG. 8, the portion marked in red is the fundamental frequency, and it can be seen that the bright band region repeatedly appears. The bright band region may be an integer multiple of the fundamental frequency. Since the frequency component of a human voice is expressed as an integer multiple of the fundamental frequency, only the fundamental frequency needs to be extracted to capture the frequency change of the voice. The extracted data may be two-dimensional data of the fundamental frequency over time.

FIG. 9 illustrates a spectrum from which noise is removed from the spectrum of FIG. 8. This is to extract feature values from the spectrum of FIG. 8, and for this purpose, it may be assumed that the recording is performed in a quiet environment. In a quiet environment, the human voice is a dominant signal (a situation where the human voice is louder than the noise), so that noise below a certain size can be ignored to remove noise. As shown in FIG. 9, only a fundamental frequency may be extracted from a spectrum from which noise is removed except for a signal corresponding to a human voice. This is the lowest band of repeated bright bands, and this one band also appears across several frequencies, so it is preferable to use the middle of them. The noise generated in this process can be solved by using a mean value and a standard deviation to create a filter that ignores signals that deviate from the mean value by more than the standard deviation.

FIG. 10 illustrates a signal spectrum obtained by extracting a fundamental frequency from the spectrum of FIG. 9. Referring to FIG. 10, it is possible to determine the presence or absence of a voice. For example, a section with a voice may be represented as a section in which the Y value is not 0, and a section without a voice may be represented as a section in which the Y value is 0. Referring to FIG. 10, it can be seen that the fundamental frequency gradually increases.

By using the two-dimensional data of the fundamental frequency of the extracted time as described above, if the section containing only the human voice can be extracted, information on the frequency of the human voice can be extracted. Once the mean value of the data is obtained, the average frequency of the voice can be calculated, which is a measure of how high the user's voice frequency band is. Use this value to find out how high your gender and voice are. The standard deviation can be used to calculate how much the frequency of the voice changes. By finding the standard deviation over the entire length of the recording, you can see how much frequency change the user made during the entire presentation. This value can be used as a measure of how boring the speaker was during the presentation.

If you cut the interval in about 20s, you can see whether the presentation was boring every 20s. It is also possible to narrow the interval to find the standard deviation. For example, a total of seven words exist in FIG. 10, and when a standard deviation is obtained for each word, it can be seen at a moment that a sudden change in frequency occurs. That is, different feature values can be extracted depending on how long the section for calculating one standard deviation is taken. For example, the frequency average value for the entire time can distinguish between the height of the average voice and the gender. The frequency standard deviation over the entire time makes it possible to check this voice telescoping change during the entire presentation. The frequency standard deviation of each section (eg, 20 s) can identify the change in height of the voice of each section. In addition, the frequency standard deviation for each node, it is possible to confirm the change in the pitch of the instantaneous voice.

To extract the values related to the amplitude of the voice, you can use RMSE, a librosa tool. RMSE is a method of grouping digital signals read at a sampling rate of 22050 Hz into specific units and then performing root mean square calculations for each unit. As in the case of STFT, the RMSE is calculated by grouping the data in 512 units, and the change in RMS value can be calculated about every 23ms. This can be used to calculate the change in size of speech over time.

11 is an energy graph of RMSE. In FIG. 11, the X axis represents time and the Y axis represents intensity. Two-dimensional data of speech can be obtained by using RMSE graph, and the two-dimensional data of speech can be checked whether or not the part of the graph pops up and the part that is not, and the Y value of the part of the voice You can calculate the size. Assuming that recording is performed in a quiet environment in order to extract feature values in FIG. 11, noise may be removed to ignore sounds below a certain size, and FIG. 12 is a graph showing main features of sound data. Referring to FIG. 12, it can be seen that a section with voice is represented by a section in which the Y value is not 0, and a section without voice is represented by a section in which the Y value is 0. FIG. You can also see how the amplitude of the voice has changed over time. When the average value of the data identified in the graph of FIG. 12 is obtained, the average size of the voice may be calculated. Using this value, you can see how loud the user's voice was during the presentation. Also, different feature values can be extracted depending on how the interval for calculating the standard deviation is taken. The standard deviation can be used as a measure of whether the loudness is loud enough, whether the presentation is boring or if there has been a sudden change in loudness. For example, the amplitude mean value over the entire time may indicate the average loudness. In addition, the amplitude standard deviation over time can represent a change in loudness during the entire presentation. In addition, the amplitude standard deviation of each section (20s) may represent a change in voice volume for each section, and the amplitude standard deviation of each node may represent a change in instantaneous voice size.

A web system and a web page for feeding back a user's speech have been implemented using the methods illustrated in FIGS. 7 to 12, which will be described in detail with reference to FIGS.

13 is a diagram showing a web system structure of the present invention.

As shown in FIG. 13, the web system of the present invention can be largely divided into a server and a client. In this case, the server may be implemented by, for example, node.js, and the client may be implemented by react.js. The server can communicate with the Google API server to get the information, and run the Python library to analyze the voice data. Thereafter, when the voice data information analyzed by the server is transmitted to the client, the information on the feedback of the voice data from the client may be displayed on the web screen.

14 is a view showing a web page according to an embodiment of the present invention.

As shown in FIG. 14, after the user uploads pre-registered voice data to a web page and presses the 'start analysis' button, the data analyzed by the system may be displayed on the web screen. The web page may display a script, a legend, a recommended video, and the like of the voice data. Script is the part that shows the result of converting to text in the Google API. It is to visually show the part of the user's speech. The legend is a button that can display 'Sudden changes in tone', 'Decibel large intervals' and 'High speed intervals' which are characteristic parts of speech in the script. The part may be displayed in the color of the button. FIG. 14 shows an example after clicking a red button of a 'speed section', and it can be seen that a word recognized as a speed section is replaced with a red word. When another legend button is clicked, a word corresponding to the legend feature can be displayed, and one legend can be displayed at a time.

In addition, the recommended video stores links to various videos suitable for men and women, horse speed, horse height, etc., analyzes the script, determines what the recorded file has, and exports the appropriate video link and links the screen. Video can be displayed.

The Google API applied for the speech feedback of the present invention predicts the accuracy to some extent when generating a speech recognition result by itself, and provides this as a value of confidence. Therefore, based on this confidence value, we can check whether the script is well calculated.

As described above, the web system and the web page for speech feedback of the present invention do not need a separate production cost, there is an advantage in terms of cost because only the cost of the server maintenance cost and Google API usage fee. In addition, since it is provided as a web service, it is advantageous compared to speech school in that it is not limited by time and space at a low cost. The web system and web page for speech feedback of the present invention provide very basic functionality free of charge and gain some revenue from advertising, while continuing to use and add functionality (e.g. storage, more professional analysis). Profit can be generated through cost structures such as premium models that require payment for use.

As mentioned above, although preferred embodiments of the present invention have been described with reference to the accompanying drawings, the present invention is not limited thereto, and those skilled in the art to which the present invention pertains may have the technical idea of the present invention. Substitutions and / or modifications can be made without departing from this scope. Therefore, the scope of the present invention should not be limited to the above-described embodiment, but should be defined by the claims below and equivalents thereof.

Claims (7)

A server that communicates with a Google API server to receive voice file information converted to text, simultaneously executes a Python library to analyze voice data, and deliver the analyzed voice data to a client device;
A display for displaying a web page configured to allow a user to check uploaded and analyzed voice file information of the voice file; And
A client that delivers a voice file to the server or delivers the voice file received from the server to the display to display on the web page.
Web service system for speech feedback comprising a.
The method of claim 1,
Python library contains librosa
Web service system for speech feedback.
The method of claim 1,
The analysis of the speech data includes frequency related feature value extraction and amplitude related feature value extraction.
Web service system for speech feedback.
The method of claim 3,
The frequency-related feature value extraction uses the short time Fourier transform (STFT) of librosa, and the amplitude-related feature value extraction uses the RMSE of librosa.
Web service system for speech feedback.
The method of claim 1,
The web page is composed of a start analysis button, a script, a legend button, a recommended video, etc.
Web service system for speech feedback.
The method of claim 5,
The script indicates the result of converting the voice file into text.
Web service system for speech feedback.
The method of claim 5,
The legend button is a button for displaying a sudden change in tone, a large decibel period, a high speed section, and the like in the script.
Web service system for speech feedback.

Images