KR20080018658A - Pronunciation comparation system for user select section - Google Patents

Abstract

A pronunciation comparison system for a user selected section is provided to increase a listening comprehension skill with respect to multimedia contents and allow a user to clearly hear his pronunciation. A storage unit(100) stores multimedia contents pronunciation. A section setting unit(105) allows a user to select a section of the multimedia contents. A noise canceling unit(130) cancels noise of the multimedia contents. An output unit(110) provides the pronunciation of a section of the multimedia contents selected by the user(110). An input unit(120) receives a pronunciation of a sentence from the user and transmits the same to the noise canceling unit(130). A pre-processing unit(140) transfers the user pronunciation of the multimedia contents to a vocal sound characteristics parameter extracting unit(160) via the noise canceling unit(130). The vocal sound characteristics parameter extracting unit(160) extracts characteristics parameter from the user voice and the pronunciation of the corresponding multimedia contents. A component characteristics parameter extracting unit(170) extracts component characteristics parameter that analyzes pitch, energy and formant components with respect to the user voice and the pronunciation of the corresponding multimedia contents. A comparing/analyzing unit(170) compares the vocal sound characteristics parameter and the component characteristics parameter by certain analysis units with respect to the user voice and the pronunciation of the corresponding multimedia contents and provides the results to the user.

Description

Speech comparison system for user selection section {PRONUNCIATION COMPARATION SYSTEM FOR USER SELECT SECTION}

1 is a block diagram of an embodiment of a voice comparison system for a user selection section according to the present invention;

2 is a screen configuration diagram of an embodiment provided to the user by the voice comparison system for the user selection section in accordance with the present invention.

3 is a user interval setting flowchart according to the present invention;

Explanation of symbols on the main parts of the drawings

100: storage means 105: section setting means

110: output means 120: input means

130: noise removing means 140: preprocessing means

150: database 160: vocal characteristics parameter extraction means

170: component characteristic parameter extraction means 175: pitch component analysis unit

180: energy component extraction unit 185: formant component extraction unit

195: comparative analysis means

The present invention relates to a voice comparison system for a user selection section. In particular, the user's pronunciation and the pronunciation of the multimedia content are removed from the user's selected multimedia content section, and the noise is output to the output means. It improves listening ability through sound source, removes noise included in user's voice, enables commercial users to clearly recognize their own voice, compares and analyzes each characteristic parameter including vocal characteristic parameter and component characteristic parameter. The present invention relates to a voice comparison system for a user selection section that enables a user to quickly and easily acquire a language with accurate pronunciation by presenting a comparative analysis result and a pronunciation correction method.

The conventional English conversation teaching materials have a problem in that the user cannot compare the pronunciation or the difference between the multimedia contents by providing only a listening function using a printed copy and a tape.

In addition, the conventional language teaching method has a problem that the learning effect is low because it is a one-way delivery method.

Accordingly, the present invention has been made to solve the above conventional problems,

An object of the present invention is to enhance the listening ability of the multimedia content by clearly outputting the user's pronunciation and the pronunciation of the multimedia content with respect to the multimedia content selected by the user, and to clearly hear the user's pronunciation. .

In addition, another object of the present invention is to present a comparison analysis result and pronunciation correction method to the user by comparing and analyzing the vocal trait and component trait parameters.

Voice comparison system for the user selection section of the present invention for achieving the above object,

In the voice comparison system,

Storage means (100) for storing the sentence to be learned and the pronunciation of the multimedia content for the sentence;

Section setting means (105) for allowing a user to select a section by dividing a part of the multimedia contents from the storage means;

Noise removing means (130) for separating the noise of the user or the multimedia contents into unvoiced and voiced sound;

An output unit 110 for providing the multimedia content pronunciation of the section of the multimedia content to be learned to the user as the section is selected by the user;

Input means (120) for receiving a pronunciation of a sentence provided by the output means from the user and transmitting the pronunciation to a noise removing means;

The starting point and the end point are detected in a sentence unit for the voice input through the input means, and each detected sentence is cut in a predetermined analysis unit and transmitted to the vocal characteristic parameter extracting means, and the sentence is input to the sentence input by the user. Preprocessing means (140) for transmitting the pronunciation of the multimedia content from said storage means to said vocal trait parameter extraction means via a noise removing means;

Voice characteristic parameter extracting means (160) for extracting characteristic parameters by classifying the cut-off user's voice and the pronunciation of the corresponding multimedia contents into silent / voiced / unvoiced sound;

Component characteristic parameter extracting means (170) for extracting component characteristic parameters obtained by analyzing pitch, energy, and formant components with respect to the cut-off user's voice and the pronunciation of the corresponding multimedia contents; And

And comparative analysis means (195) for comparing and analyzing the vowel characteristic parameter and the component characteristic parameter for the pronunciation of the user's voice and the corresponding multimedia contents for each predetermined analysis unit to provide a comparative analysis result to the user. do.

The predetermined analysis unit of the preprocessing means is

According to the user's selection it is characterized in that the word, word, or sentence unit.

The noise removing means,

The method of classifying voiced sound, unvoiced sound and silence using the wavelet-converted signal as a parameter on the spectrum is used.

The pretreatment means,

After separating the frequency band of the voice into three bands in consideration of the human hearing characteristics, it is characterized in that the way to search for the end point of the voice by setting a fine energy threshold value for each band.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an embodiment of a voice comparison system for a user selection section according to the present invention.

As shown in Figure 1, the present invention, the voice comparison system for the user selection section,

In the voice comparison system,

Storage means (100) for storing the sentence to be learned and the pronunciation of the multimedia content for the sentence;

Section setting means (105) for allowing a user to select a section by dividing a part of the multimedia contents from the storage means;

Noise removing means 130 for removing the noise of the user or multimedia content divided into unvoiced and voiced sound;

Outputting means (110) for providing pronunciation of multimedia contents of the section of the multimedia content to be learned to the user as the user selects the section;

Input means (120) for receiving a pronunciation of a sentence provided by the output means from the user and transmitting the pronunciation to a noise removing means;

The starting point and the end point are detected in a sentence unit for the voice input through the input means, and each detected sentence is cut in a predetermined analysis unit and transmitted to the vocal characteristic parameter extracting means, and the sentence is input to the sentence input by the user. Preprocessing means (140) for transmitting the pronunciation of the multimedia content from said storage means to said vocal trait parameter extraction means via a noise removing means;

Voice characteristic parameter extracting means (160) for extracting characteristic parameters by classifying the cut-off user's voice and the pronunciation of the corresponding multimedia contents into silent / voiced / unvoiced sound;

Component characteristic parameter extracting means (170) for extracting component characteristic parameters obtained by analyzing pitch, energy, and formant components with respect to the cut-off user's voice and the pronunciation of the corresponding multimedia contents; And

And comparative analysis means (195) for comparing and analyzing vowel characteristic parameters and component characteristic parameters for the pronunciation of the user's voice and the corresponding multimedia contents for each predetermined analysis unit to provide a comparative analysis result to the user. All.

The storage means 100 is a means for storing the learning target sentence and the pronunciation of the multimedia content for the sentence.

The section setting means 105 is a means for allowing a user to select a section by dividing a part of the multimedia content from the storage means 100.

For example, the section setting means can be set in units of time, words, and sentences.

3 is a flowchart of the section setting means.

Before setting the section, the user inputs and inputs the section setting parameter of the user in advance (S310).

The section setting parameter is a method of setting the time such as 1 second, 2 seconds, 5 seconds, 10 seconds, the method of setting the number of words such as 1 word, 2 words, 3 words, the method of setting sentences such as 1 sentence, 2 sentences, etc. This can be.

If the time is set, the user selects the section start point and automatically selects the point that has been advanced during the corresponding time from the start point as the section end point.

When the number of words is set, the end point is designated as the number of words advanced from the starting point.

When the number of sentences is set, the end point is designated as the number of sentences advanced from the starting point.

When the section is selected through the above steps, the start point and the end point are extracted as described above (S320). After the section selection, the text of the multimedia content of the selected section is output on the screen (S330).

The section selection may include a process of determining whether waveform comparison is possible (S340).

In other words, if the multimedia contents correspond to only sound or sound effects, not human pronunciation, such a case is not selected because the comparison between human voice and waveform is impossible.

The output means 110 provides the user with the sentence to be learned and the pronunciation of the multimedia content of the sentence according to the user's selection, and outputs the user's input voice upon the user's request.

Then, the section setting means 105 provides the user with the sentence stored in the database 150 through the output means (110).

The input means 120 (for example, a microphone, etc.) for inputting a user's voice is used by the user while watching the English conversation contents displayed in a text format on the upper portion 210 of the screen through the output means 110. In order to compare the voice of the multimedia content with the voice, the user's voice is input through the microphone using the record button.

The noise removing unit 130 is a means for dividing the noise of the user or the multimedia content into unvoiced voices and voiced voices, and processes the noise before outputting it to the output means to improve the listening power by outputting a voice without noise to the user.

In the present invention, the noise removing means performs noise processing by using voiced / unvoiced separation. Voiced sound and unvoiced sound are an important feature of speech. Instead of using the same noise processing techniques for voiced and unvoiced parts, noise processing is performed by considering their characteristics. The separation of voiced sound / unvoiced sound can be obtained using zero crossing rate and energy, and a modified voice / noise predominance determination method can be used based on voiced / unvoiced sound separation information. This change is used as a parameter in the spectrum to classify voiced / unvoiced / silent.

The method used has a good conclusion as a result of performance evaluation on the speech sentences contaminated with white noise and airplane noise.

Then, the segmental signal-to-noise ratio is obtained for the sentences contaminated with various input signal-to-noise ratios (SNR). These techniques are described in “Noise Processing Using Voiced / Unvoiced Separation” (Korean Journal of the Acoustical Society of Korea, Vol. 21, 2002) and “Voice / Unvoiced / Silent Classification of Speech Signals Using Wavelet Transformation” (Korean Society for the Acoustical Society of Korea, 1998). It is described in detail.

The preprocessing means 140 detects a start point and an end point in units of sentences with respect to the voice input through the input unit, cuts each of the detected sentences in a predetermined analysis unit, and transmits them to the vocal characteristic parameter extracting means. It is a means for transferring the pronunciation of the multimedia content of the section selected by the user for the sentence input by the voice from the storage means to the vocal characteristic parameter through the noise removing means.

The preprocessing means 140 performs a function of comparing the user's voice input / recorded through the user's voice input means 120 with the voice of the multimedia content previously stored in the database 150, and also adjusting the gain. Through the voice input from the user without distortion.

The predetermined analysis unit of the preprocessing means is

According to the user's selection it is characterized in that the word, word, or sentence unit.

In addition, the preprocessing means 140 constantly cuts the voice to analyze the characteristics of the voice. The start / end point detection technique applied to the process finds only the section in which the user speaks among the signals input through the user voice input means. As a necessary technique, if this technique is applied, the user simply speaks when desired without any operation, and automatically extracts and reproduces the voice section.

One of the problems to be solved first of all for the practical use of the present invention is detection of a start point / end point in a noise environment.

In the absence of noise, the existing energy parameters can detect the start / end point to some extent, but most of the results are not good under the actual noise environment such as urban noise.

As a method of removing background noise in an urban environment, a spectral subtraction method based on a bark scale, which is a preprocessing technique that removes only a magnitude component of a speech spectrum damaged by ambient noise, may be used. Since this is done through the noise removing means 130, it can be regarded as a process after the noise is removed.

The preprocessing means 140 of the present invention adopts a method of searching for an end point of the voice by separating the frequency band of the voice into three bands in consideration of the human auditory characteristics, and setting a fine energy threshold value for each band. In order to verify the effectiveness of the proposed method, starting / endpoint detection was performed using a database recorded in a noisy environment such as an office or a subway station. As a result, the average error rate was reduced by 46% compared to the method using the energy and zero crossing rate. Compared with energy only, the average error rate was reduced by 17%, confirming the effectiveness of the proposed method.

The preprocessing means 140 of the present invention can be seen in more detail in the "end point detection algorithm of noise speech using band energy" (Korean Society for Acoustics, 2002, Park Ki-Sang; Seok Soo-Young; Jeong Ho-Yeol;

In addition, the preprocessing unit 140 uses DTW (Dynamic Time Warping) technology to match the length and pattern of the two voices before comparing the pronunciation similarity between the user and the multimedia content.

Pattern matching uses DTW (Dynamic Time Warping) technology to match the length and pattern of the two voices before comparing the pronunciation similarity between the user and the multimedia content.

For example, if the pronunciation sentence of the multimedia content is 5 seconds, but the user pronounces the same sentence within 3 seconds because of the pronunciation of the speaker too quickly, simply comparing the sentence as it is, even if the pronunciation is incorrect, the sentence comparison is bad. Before, use DTW techniques to properly match the lengths of the two sentences.

In addition, in the analysis using DTW (Dynamic Time Warping), the characteristic parameters obtained from the vocal trait parameter extraction means and the component characteristic parameter extraction means also match the length and pattern through the DTW algorithm, and measure the similarity between the two sentences.

Pattern matching may use an algorithm for detecting a speech interval and pitch for continuous speech recognition.

This is a method of inputting continuous voice to classify the vowel / vowels by frame unit and detecting the pitch from the divided voiced voices. It is possible to extract the robust voice section in the noisy environment by inputting the voice in the real noise environment and using the appropriate threshold energy. An algorithm using zero crossing rate and short section energy in frame unit in the extracted speech section detects the pitch of voiced sounds and separates the vowels and vowels. "Study on speech section and pitch detection for continuous speech recognition" (Korean Journal of Multimedia Research, 2005, Kim, Tae-Seok; Jang, Jong-Chil).

The vocal characteristic parameter extracting means 160 is a means for extracting characteristic parameters by classifying the cut-off user's voice and the pronunciation of the corresponding multimedia content as silent / voiced / unvoiced.

In general, voice signals are classified into three types according to the characteristics of the waveform: quasi-periodic voiced sounds, unvoiced sounds similar to noise, and silence corresponding to background noise without periodicity.

In the existing voiced / unvoiced / silent classification methods, pitch information, energy, and zero crossing rate are widely used as parameters for classification.

In the present invention, a voiced / unvoiced / silent classification algorithm using the change in the spectrum as a parameter in the wavelet-converted signal is used.

This technique is described in “voiced / unvoiced / silent classification of speech signal using wavelet transform” (Korean Society for Acoustics Society, 1998).

The component characteristic parameter extracting unit 170 extracts the component characteristic parameter of the voice in order to compare the pronunciation of the user's input voice (pronunciation) and the multimedia content. Here, the component characteristic parameters include pitch, energy, and formant components.

First, the pitch component analyzer 175 analyzes a user's voice (pronunciation) and compares the height, length, stress, and intonation of the pronounced sentence to the user, thereby allowing the user to identify a problem of his / her pronunciation. . Here, the pitch is gradually changed according to the height and speed of the sound in the speech section. The present invention uses the modified AMDF method including the AMDF method and the median smoothing process among many algorithms for estimating the pitch parameter.

The energy component extractor 180 obtains the energy of the speech signal, where the energy allows the user to know which part of the pronounced sentence is emphasized and assists the results of the pitch analysis. The process of extracting energy components is performed by finding the mean square for each analysis frame.

The formant component extractor 185 extracts a formant component, which is a component characteristic parameter used to compare pronunciation of a word. In the case of short words, it is very difficult to measure the similarity with the pitch information, and thus formant information representing the frequency characteristic of the voice is used. Formants are obtained frame by frame using the peak-picking method.

Human voices have unique characteristics depending on the gender of the speaker as well as information on meaning.

That is, the voice may be classified into a high female voice and a male voice. However, in the existing speech recognition system using the HMM, although there are such characteristics of male and female voices, it does not consider this and constitutes one HMM.

As a result of the experiment, it was found that the formant frequency of male and female was 100 ~ 30Hz difference, and considering the characteristics, a method to distinguish male and female voices should be adopted.

In addition, the GMM is trained by dividing the male and female voices separately, and according to the formant characteristics of the input voice in the recognition process, the male voice is recognized as the male HMM and the female voice as the female HMM. The 5.2% female voice shows a 4.4% improvement. Formant component extraction is described in detail in "Design and Implementation of a Speech Recognition Preprocessing System Using Formant Frequency" (Korean Information Science Society: 1999, Tae-Wook Kim et al.).

The comparison analysis means 195 compares the voice characteristic parameters obtained through the above-described voice signal processing process and provides audio-visually how similar the user input voice (pronunciation) and the voice of the multimedia content (pronunciation) are.

Here, the provided content displays characteristic parameters such as voiced voice, unvoiced sound, pitch, energy, formant, etc. of the two voices through the user screen 250 so as to be easily visible, and indicates the pronunciation of the parts having a large difference.

In addition, the dialog box 280 at the bottom of the screen describes the problem of user pronunciation in a text form to enable correction.

The user grasps the problem of his / her pronunciation based on the result of the evaluation result and the correction method in the comparative analysis means 195, and repeatedly corrects the pronunciation while performing the correction method suggested by the evaluation result and the correction method.

The database 150 provides a criterion for comparing the accuracy of the pronunciation of the user. The database 150 stores the pronunciation of the multimedia content in advance, and provides the user's voice input through the user's voice input means and sentences provided as learning materials. Save it.

2 is an exemplary screen configuration provided to a user by a voice comparison system for a user selection section according to the present invention.

The part 210 for providing the English sentence to the user so that the user can pronounce,

Function key 220 to listen to the user's pronunciation and the pronunciation of the multimedia content built in the database, function key 230 to allow the user to input voice, pitch of two pronunciations, energy, formants, etc. Function key 240 for visually viewing the measurement result of the pronunciation accuracy by comparing the speech characteristic parameters, part 250 for providing the user with the measurement result of the pronunciation accuracy provided to the user by the "240", multimedia A function key 260 for adjusting the pronunciation speed of the content, a function key 270 for presenting the pronunciation correction method in text, and a portion 280 in which the pronunciation correction method presented by "270" is presented Represents a function key 290 that allows the unit to be selected as a whole word / word / sentence when comparing pronunciation.

The voice comparison system for the user selection section described above may be implemented by a single system or built as a server to provide a language learning service to a user through a communication network such as the Internet.

Those skilled in the art to which the present invention pertains as described above may understand that the present invention may be implemented in other specific forms without changing the technical spirit or essential features of the present invention. Therefore, the above-described embodiments are to be understood as illustrative in all respects and not restrictive.

The scope of the invention is indicated by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the invention. do.

As described above, the voice comparison system for the user selection section of the present invention,

By applying the audio signal processing technology and the multimedia functions of the computer, the user's listening ability is enhanced by providing the user with his / her own pronunciation and the pronunciation of the multimedia contents and presenting distinct pronunciation with noise removed.

It provides audio-visual comparison results between pronunciation of one's own pronunciation and multimedia contents, and corrects the pronunciation according to the result, and maximizes the learning effect due to repeated learning.

In addition, by linking with the online learning network, it is possible to learn languages by interacting with instructors and students in real time, thereby improving users' English learning effect and usability, and investing enormous expenditure for English conversation education. Minimizing and dealing with the absolute lack of English speaking instructors can increase the breadth of education.

Claims (5)

In the voice comparison system, Storage means for storing a sentence to be learned and the pronunciation of the multimedia content of the sentence; Section setting means for allowing a user to select a section by dividing a part of the multimedia contents from the storage means; Noise removing means for dividing the noise of the user or the multimedia contents into unvoiced and voiced sound; Output means for providing a pronunciation of multimedia content of a section of the multimedia content to be learned to the user as the section is selected by the user; Input means for receiving a pronunciation of a sentence provided by the output means from the user and transmitting the pronunciation to a noise removing means;  The starting point and the end point are detected in a sentence unit for the voice input through the input means, and each detected sentence is cut in a predetermined analysis unit and transmitted to the vocal characteristic parameter extracting means, Preprocessing means for transferring the pronunciation of the multimedia content from the storage means to the vocal trait parameter extraction means via the noise removing means; Voice sound feature parameter extracting means for extracting the feature parameter by classifying the cut-off user's voice and the pronunciation of the corresponding multimedia content into silent / voiced / unvoiced sound; Component characteristic parameter extracting means for extracting component characteristic parameters obtained by analyzing pitch, energy, and formant components with respect to the cut-off user's voice and the pronunciation of the corresponding multimedia contents; And And comparative analysis means for comparing and analyzing the vowel characteristic parameter and the component characteristic parameter for the pronunciation of the user's voice and the corresponding multimedia contents for each predetermined analysis unit to provide a comparative analysis result to the user. Voice comparison system for user selection section. The method of claim 1, The predetermined analysis unit of the preprocessing means is Voice comparison system for the user selection section, characterized in that the unit of words, words, or sentences according to the user's selection. The method of claim 1, The noise removing means, A voice comparison system for a user selection section using a method of classifying voiced sound, unvoiced sound, and silence by using the wavelet transformed signal as a parameter on a spectrum. The method of claim 1, The pretreatment means, A speech comparison system for a user selection section characterized in that the frequency band of the speech is divided into three bands in consideration of human hearing characteristics, and then a fine energy threshold value is set for each band to search for an end point of the speech. The method of claim 1, The section setting means is a voice comparison system for a user selection section, characterized in that can be set in units of time, words, sentences.

Images