KR102042344B1 - Apparatus for judging the similiarity between voices and the method for judging the similiarity between voices - Google Patents

Abstract

The present invention relates to a voice similarity determining device and a voice similarity determining method capable of determining and providing a similarity between a reference voice extracted from an image file and a test voice to be scored.

Description

Apparatus for judging the similiarity between voices and the method for judging the similiarity between voices}

The present invention relates to an apparatus for determining a speech similarity and a method for determining the speech similarity, and more specifically, to determine and score a similarity between a reference speech signal and a test speech signal and provide the result to the learner, thereby providing an objective reflection on the learning result. The invention is characterized in that it can provide motivation to improve the learning efficiency of the learner through.

Shadowing, which learns to speak or listen to the sound almost at the same time, then repeats and learns, is more than simply listening in two processes, where the learner hears the material and then listens to it. Much effort has been made to remember information longer. In other words, when shadowing, learners can concentrate more on processing and information than simple listening, and the burden of speech can be reduced by following the speaker's words, and it becomes possible to pay attention to the language itself, thereby increasing learning efficiency. It works. Therefore, many experts recommend such shadowing, which is understood and spoken at the same time, as a foreign language learning method.

1 is a view showing the state of the smart phone.

A smart phone is a terminal device 10 having a function similar to a personal computer (PC). The smart phone is a mobile phone which combines a conventional mobile phone function with a personal information management function and a wireless Internet service function. Refers to. The user can install and use a variety of applications (applications) on these smartphones, and in recent years, they have utilized a lot of these smartphones for learning foreign languages. That is, many applications that can practice speaking, listening, and writing for language learning such as foreign languages are provided, and an application having a recording function for shadowing learning is also provided.

However, these existing applications have a problem in that they cannot provide accurate feedback on the shadowing learning by providing only a recording function or an evaluation function such as pass or nonpass. .

The present invention has been made to solve the above problems, the voice similarity determination method that can provide an objective learning result to the user by providing a score by comparing the similarity between the reference voice signal and the test voice signal uttered by him and An apparatus for determining a voice similarity is provided.

The present invention also provides a voice similarity determining method and a voice similarity determining apparatus capable of determining similarity in terms of accent as well as pronunciation.

In order to achieve the above object, the present invention comprises the steps of calculating the first speech length in the reference speech signal; Extracting first mel frequency cepstal coefficients (mfcc) from the reference speech signal; Calculating a second speech length in a test speech signal; Extracting second MFCs from the test voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating dynamic time warping (dtw) scores between the first MFCs and the second MFCs; Mapping the decoded unique scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio.

In addition, the present invention comprises the steps of receiving the first voice length and the first MFC of the reference voice signal from an external server (server); Calculating a second speech length in the test speech signal; Extracting second MFCs from the test voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating detired double scores between the first MFCs and the second MFCs; Converting the decodeable scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio.

In addition, the present invention comprises the steps of receiving the second voice length and the second MFC of the test voice signal from the terminal device; Calculating a first speech length from the reference speech signal; Extracting first MFCs from the reference voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating detired double scores between the first MFCs and the second MFCs; Converting the decodeable scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio.

In addition, the present invention comprises the steps of receiving the second voice length and the second MFC of the test voice signal from the terminal device; Receiving a first voice length and a first MFC from a reference voice database; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating detired double scores between the first MFCs and the second MFCs; Converting the decodeable scores into mapping scores through mapping; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio.

In addition, the weight function has a weight value of 1 when the speech length ratio is 1, and has a weight value of 0 when the speech length ratio is 0.5 or less and 1.5 or more.

In addition, the weight function has a slope value of 2 in a section in which the speech length ratio is 0.5 or more and 1 or less, and a slope value of -2 in a section in which the speech length ratio is 1 or more and 1.5 or less. To provide.

In addition, after the accent score is calculated by multiplying the pronunciation score by the weighted speech length ratio, calculating a weighted pronunciation score by multiplying the mapping score by a pronunciation weight; Calculating a weighted accent score by multiplying the accent score by an accent weight; And calculating the final similarity score by adding the weighted pronunciation score and the weight accent score to provide a voice similarity determining method.

In addition, the pronunciation weight is 0.52, the accent weight provides a voice similarity determining method, characterized in that 0.48.

In addition, in the step of calculating the debitable U scores between the first MCs and the second FM, the deutable double scores are obtained by using the first and the second CMs. Provided is a voice similarity determination method characterized in that it is obtained by adding values on a path in a prepared detittable distance matrix.

In addition, the step of converting the deddable U scores are converted into a mapping score may be the minimum value of the deddable U scores to the maximum score, and the maximum value of the Detti double scores to the minimum score

It is converted into a mapping score using an equation, and the detired oil score is obtained by equalizing the optimal path length in the detired oil distance matrix with respect to the length of the optimal path. Provided is a voice similarity determining method, which is set in advance.

The method may further include removing noise and / or a background sound from the reference voice signal and / or the test voice signal before the first voice length is calculated from the reference voice signal. Provide a method.

The method may further include removing noise and / or a background sound from the test voice signal before the second voice length is calculated from the test voice signal.

The method may further include removing noise and / or a background sound from the reference voice signal before calculating the first voice length from the reference voice signal.

In addition, the removal of noise and / or background sound provides a method for determining the voice similarity, characterized in that through the Winner filter (winer) filter.

In addition, the first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and DFT (DFT, Discrete Fourier) sequentially. Transform), Mel scale filter bank, and Discrete Cosine Transform (DCT) are applied, and among them, CEM (Cepstral Mean Normalization) It provides a voice similarity determination method characterized in that extracted by applying.

In addition, the present invention includes the steps of calculating the second voice length in the test voice signal; Extracting a second MC from the second voice interval signal; The second voice length and the second MFC is transmitted to an external server; provides a voice similarity determination method comprising a.

The present invention also includes a first speech interval detection module for calculating a first speech length from a reference speech signal; A second voice interval detection module for calculating a second voice length from the test voice signal; The first voice length is received from the first voice interval detection module, the second voice length is received from the second voice interval detection module, and the first voice length is divided by the second voice length to speak length ratio. Ignition length ratio calculation module for calculating; A first MC extracting module for extracting first MCs from the reference speech signal; A second MC extracting module for extracting second MCs from the test voice signal; A decoded-deowable module receiving the first MCs from the first MC extracting module and receiving the second MCs from the second MC extracting module to calculate detired oil scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping.

In addition, the present invention includes a storage module for receiving and storing the first voice length and the first MFC of the reference voice signal from an external server; A second voice interval detection module for calculating a second voice length from the test voice signal; An utterance length for receiving the second voice length from the second voice interval detection module and receiving a first voice length from the storage module and dividing the first voice length by the second voice length to calculate a utterance length ratio Non-calculating module; A second MC extracting module for extracting second MCs from the test voice signal; A DTI double U module receiving first MFCs from the storage module, and receiving second MFCs from the second MFC module to calculate detired double scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping.

In addition, the present invention includes a storage module for receiving and storing the second voice length and the second MFC of the test voice from the terminal device; A first voice interval detection module configured to calculate a first voice length from a reference voice signal; A utterance length for receiving a first voice length from the first voice interval detection module and receiving a second voice length from the storage module and dividing the first voice length by the second voice length to calculate a speech length ratio. Non-calculating module; A first MC extracting module for extracting first MCs from the reference speech signal; A DTI double U module receiving second MFCs from the storage module, and receiving first MFCs from the first MFC module to calculate detired double scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping.

In addition, the present invention includes a storage module for receiving and storing the second voice length and the second MFC of the test voice from the terminal device; A reference voice database storing the first voice length of the reference voice and the first MFCs; Speech length ratio calculation for receiving the first speech length from the reference speech database, receiving the second speech length from the storage module, and calculating the speech length ratio by dividing the first speech length by the second speech length. module; A deddable-you module for receiving the first MFCs from the reference voice database, receiving the second MFCs from the storage module, and calculating a decoded-U scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping.

In addition, the weight function has a weight value of 1 when the speech length ratio is 1, and has a weight value of 0 when the speech length ratio is 0.5 or less and 1.5 or more.

In addition, the weight function has a slope value of 2 in a section in which the speech length ratio is 0.5 or more and 1 or less, and a slope value of -2 in a section in which the speech length ratio is 1 or more and 1.5 or less. To provide.

In addition, a weighted pronunciation score is calculated by multiplying the pronunciation score received from the pronunciation score calculation module, multiplying the accent score received from the accent score calculation module, and calculating a weighted accent score by multiplying the accent score. And a final similarity score calculation module that adds a weighted pronunciation score and the weight accent score to calculate a final similarity score.

In addition, the pronunciation weight is 0.52, and the accent weight is provided with a voice similarity determination device, characterized in that 0.48.

In addition, the D TW module calculates the D TD scores by adding values on a path in a D T double distance matrix generated by using the first Ms and the second Ms. Provide the device.

In addition, the mapping score used by the intonation score calculation module and / or the pronunciation score calculation module is the minimum value of the deddableness points received from the decoded use module as the maximum score, and the maximum value of the decodedness points. With the minimum score

It is converted into the mapping score by using an equation, and the detired oil score is obtained by equalizing the optimal path length in the detired oil distance matrix with respect to the length of the optimal path. Provides a voice similarity device which is predetermined.

In addition, the first filter module to remove the noise and / or background sound from the reference voice signal to pass to the first voice interval detection module; And a second filter module which removes noise and / or background sound from the test voice signal and delivers the noise to the second voice section detection module.

The apparatus may further include a second filter module configured to remove noise and / or background noise from the test voice signal and transmit the noise and / or background sound to the second voice interval detection module.

The apparatus may further include a second filter module which removes noise and / or background sound from the reference voice signal and transmits the noise to the second voice interval detection module.

In addition, the first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and DFT (DFT, Discrete Fourier) sequentially. Transform), Mel scale filter bank, and Discrete Cosine Transform (DCT) are applied, and among them, CEM (Cepstral Mean Normalization) Provided is a speech similarity determining apparatus, which is extracted by applying.

In addition, the present invention comprises the steps of calculating the first speech length in the reference speech signal; Extracting first mel frequency cepstal coefficients (mfcc) from the reference speech signal; Calculating a second speech length in a test speech signal; Extracting second MFCs from the test voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating dynamic time warping (dtw) scores between the first MFCs and the second MFCs; Mapping the decoded unique scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio, wherein the first speech length and / or the second speech length include energy, zero crossing rate, autocorrelation coefficient, increase / decrease coefficient, and first order Elp. Provided is a voice similarity determination method characterized in that it is detected using any one or more of (lpc, linear predictive coding) coefficient, predictive error, differential energy, snr (siganl to noise ratio).

In addition, the present invention comprises the steps of receiving the first voice length and the first MFC of the reference voice signal from an external server (server); Calculating a second speech length in the test speech signal; Extracting second MFCs from the test voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating detired double scores between the first MFCs and the second MFCs; Converting the decodeable scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio, wherein the first speech length and / or the second speech length include energy, zero crossing rate, autocorrelation coefficient, increase / decrease coefficient, and first order Elp. Provided is a voice similarity determination method characterized in that it is detected using any one or more of the seed coefficient, predictive error, differential energy, SNR.

In addition, the present invention comprises the steps of receiving the second voice length and the second MFC of the test voice signal from the terminal device; Calculating a first speech length from the reference speech signal; Extracting first MFCs from the reference voice signal; Calculating a speech length ratio by dividing the first voice length by the second voice length; Calculating detired double scores between the first MFCs and the second MFCs; Converting the decodeable scores into a mapping score; Calculating a weighted speech length ratio by applying a weight function to the speech length ratio; And calculating the accent score by multiplying the mapping score by the weighted speech length ratio, wherein the first speech length and / or the second speech length include energy, zero crossing rate, autocorrelation coefficient, increase / decrease coefficient, and first order Elp. Provided is a voice similarity determination method characterized in that it is detected using any one or more of the seed coefficient, predictive error, differential energy, SNR.

The present invention also includes a first speech interval detection module for calculating a first speech length from a reference speech signal; A second voice interval detection module for calculating a second voice length from the test voice signal; The first voice length is received from the first voice interval detection module, the second voice length is received from the second voice interval detection module, and the first voice length is divided by the second voice length to speak length ratio. Ignition length ratio calculation module for calculating; A first MC extracting module for extracting first MCs from the reference speech signal; A second MC extracting module for extracting second MCs from the test voice signal; A decoded-deowable module receiving the first MCs from the first MC extracting module and receiving the second MCs from the second MC extracting module to calculate detired oil scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping. The first voice interval detection module may include energy, The first speech length is calculated using one or more of zero crossing rate, autocorrelation coefficient, increase and decrease coefficient, first order LPC coefficient, prediction error, differential energy, and snr (siganl to noise ratio), and the second The voice interval detection module uses one or more of energy, zero crossing rate, autocorrelation coefficient, increase and decrease coefficient, first order LPC coefficient, prediction error, differential energy, and snr (siganl to noise ratio). Provided is a voice similarity determining device, characterized in that the two voice length is calculated.

In addition, the present invention includes a storage module for receiving and storing the first voice length and the first MFC of the reference voice signal from an external server; A second voice interval detection module for calculating a second voice length from the test voice signal; An utterance length for receiving the second voice length from the second voice interval detection module and receiving a first voice length from the storage module and dividing the first voice length by the second voice length to calculate a utterance length ratio Non-calculating module; A second MC extracting module for extracting second MCs from the test voice signal; A DTI double U module receiving first MFCs from the storage module, and receiving second MFCs from the second MFC module to calculate detired double scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping. The second voice interval detection module includes energy, zero crossing rate, Provided is a speech similarity determining apparatus, characterized in that the second speech interval length is calculated using at least one of autocorrelation coefficient, increase and decrease coefficient, first order LPC coefficient, prediction error, differential energy, and SNL.

In addition, the present invention includes a storage module for receiving and storing the second voice length and the second MFC of the test voice from the terminal device; A first voice interval detection module configured to calculate a first voice length from a reference voice signal; A utterance length for receiving a first voice length from the first voice interval detection module and receiving a second voice length from the storage module and dividing the first voice length by the second voice length to calculate a speech length ratio. Non-calculating module; A first MC extracting module for extracting first MCs from the reference speech signal; A DTI double U module receiving second MFCs from the storage module, and receiving first MFCs from the first MFC module to calculate detired double scores; The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio; And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping. The first voice interval detection module may include energy, zero crossing rate, Voice similarity judgment, characterized in that the first speech length is calculated using one or more of autocorrelation coefficient, increase and decrease coefficient, first order LPC coefficient, prediction error, differential energy, and snr (siganl to noise ratio) Provide the device.

According to the voice similarity determining method and the voice similarity determining apparatus according to the present invention, the following effects are obtained.

First, the similarity between the reference voice and the test voice heard and followed is scored and feedbacked, thereby improving the learning efficiency of the learner. In addition, the effect of providing motivation for learning can be obtained.

Second, since the score for the accent as well as the pronunciation is calculated and provided, it is possible to compare the difference between the test voice and the reference voice spoken by the user from various angles. That is, there is a high correlation between the pronunciation characteristics and the intonation characteristics of the person, there is an effect that can provide information about the intonation characteristics by using the MFCs and the speech length ratio indicating the pronunciation characteristics.

Third, the similarity determination method between the reference voice and the test voice according to the present invention can be extended to the speech recognition field in the future.

Fourth, according to the embodiment in which the work division between the external server and the terminal device is implemented, the load generated during the preprocessing and the similarity determination process of the voice signal can be distributed and reduced. In addition, according to an example in which the first voice length of the reference voice signal and the first MFC are databased and implemented, the similarity determination calculation between the reference voice signal and the test voice signal can be quickly and effectively scored and provided. Can be achieved.

Fifth, the voice similarity determining apparatus and the voice similarity determining method according to the present invention have an effect of enabling a more efficient similarity analysis by processing a voice signal received through a mel scale filter bank reflecting the hearing characteristics of a person.

Sixth, the apparatus for determining the voice similarity according to the present invention may be used as an objective basis for whether the user's pronunciation is improved by scoring and providing an evaluation for pronunciation or intonation. In addition, by accumulating and storing such score data, there is an effect that can be provided to analyze the portion of the strengths and weaknesses in the user's language.

1 is a view showing the state of the smart phone.
2 is a block diagram illustrating an internal configuration of an apparatus for determining voice similarity according to the present invention.
3 is a diagram illustrating a situation in which the voice similarity determining apparatus according to the present invention may be modified.
4 is a view showing a modified embodiment of the voice similarity determining apparatus according to the present invention.
5 is a view showing another modified embodiment of the voice similarity determining apparatus according to the present invention.
6 is a view showing another modified embodiment of the voice similarity determining apparatus according to the present invention.
7 is a block diagram illustrating an example of a filter module of an apparatus for determining voice similarity according to the present invention.
8 is a diagram illustrating voice waveforms and spectrograms before and after Wiener filtering.
9 is a diagram illustrating an example of a speech section detection module of the apparatus for determining speech similarity according to the present invention.
10 is a view showing another example of the voice interval detection module of the voice similarity determination apparatus according to the present invention.
11 is a block diagram showing the internal configuration of the MFC extraction module of the voice similarity determination apparatus according to the present invention.
12 is a diagram illustrating a process of extracting a feature from a voice signal by moving a frame.
13 is a view showing the shape of the hamming window.
FIG. 14 is a view showing a triangular filter bank provided through a mel scale filter bank section.
FIG. 15 is a diagram illustrating a process of calculating detired double scores made through the detired double module of the voice similarity determining apparatus of the present invention.
FIG. 16 is a view illustrating a process of converting a decoded U score into a mapping score through a pronunciation score calculation module and / or an accent score calculation module of the present invention speech similarity determination device.
17 is a view showing the weight function used in the intonation score calculation module of the present invention speech similarity determination apparatus.
18 is a flowchart illustrating a processing procedure of a voice similarity determining method according to the present invention.
19 is a flowchart illustrating a modified embodiment of the voice similarity determining method according to the present invention.
20 is a flowchart illustrating another embodiment of a method for determining speech similarity according to the present invention.
FIG. 21 is a flowchart illustrating still another embodiment of the present invention.
22 is a flowchart illustrating a process of transmitting the second voice length and the second MFCs from the terminal device configuring the voice similarity determination device according to the present invention to an external server.
FIG. 23 is a table showing correlations between score results calculated according to a voice similarity determining apparatus and a voice similarity determining method according to the present invention and score results evaluated by expert scorers.
FIG. 24 is a table showing correlations between scores calculated by using a pronunciation score and an accent score calculated through the present invention in a linear regression model and score results evaluated by expert scorers.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

In addition, in the description of the present invention, the terms "first", "second", and the like are used only for the purpose of distinguishing one component from other components, and are not used to limit any meaning. And the singular forms “a”, “an” and “the” include plural expressions unless the context clearly dictates otherwise, and the terms “comprise” or “having” may include features, numbers, steps, operations, components, parts, or parts described in the specification. It is to be understood that combinations of these are intended to exist, and do not preclude the existence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof in advance.

2 is a block diagram illustrating an internal configuration of an apparatus for determining voice similarity according to the present invention.

The apparatus for determining voice similarity according to the present invention includes a part for processing a reference voice signal largely included in an image, a part for processing a test voice signal which is a voice spoken by a user, and a part for determining and scoring a similarity between two voice signals. It can be configured to include. More specifically, the part processing the reference voice signal may include extracting a first filter module 111, a first voice interval detection module 121, and a first mel frequency ceptrum coefficient (mfcc). The module 131 may be configured, and the part for processing the test voice signal may include a second filter module 112, a second voice interval detection module 122, and a second MC extract module 132. Can be configured. Here, the first filter module 111 and the second filter module 112, the first voice interval detection module 121 and the second voice interval detection module 122, the first MC extract module 131 and the first 2 The MC extracting module 132 has the same internal structure or role, but in terms of receiving different signals and processing them, the modifier 'first' is used for a module that processes the reference voice signal for convenience of distinction. Next, the module that processes the test voice signal is denoted by the second modifier. In addition, a part of determining and scoring the similarity between the voice signals may include a dynamic time warping (dtw) module 150, a speech length ratio calculating module 140, a pronunciation score calculating module 160, and an accent scoring calculation module ( 170, the final similarity score calculation module 180 may be configured.

More specifically, the noise and / or the background sound of the reference voice signal may be removed by passing through the first filter module 111, and the voice section and the silent section included in the reference voice signal may be removed from the first voice section detection module ( 121, it may be recognized by distinguishing which part is a voice section or a silent section. Then, the first MFCs representing the characteristics of the reference voice signal may be extracted by passing through the first MFC extraction module 131. In the test voice signal, the process of removing noise and / or background sound, extracting the second MFCs after the speech section and the silent section are recognized may be performed through the same process. The second voice section detection module 122 and the second MFC extraction module 132 may be made. Detailed operation of each module will be described in more detail with reference to the following drawings.

The process of determining the similarity between the reference voice signal and the test voice signal may be performed through the following process.

First, the first voice length calculated through the first voice interval detection module 121 and the second voice length calculated through the second voice interval detection module 122 are transmitted to the speech length ratio calculation module 140. The received speech length ratio calculation module 140 may perform a process of calculating the speech length ratio by dividing the first speech length by the second speech length. In addition, the DMF module 150 that receives the first MFCs extracted by the first MFC extraction module 131 and the second MFCs extracted by the second MFC extraction module 132 is a DTI double oil. The scores of the scores may be calculated, and the scores of the TTIs calculated by the DTI module 150 may be transferred to the pronunciation score calculator 160 to be converted into pronunciation scores through a mapping process. have. In addition, the intonation score calculation module 170 calculates the weighted speech length ratio by applying a weighting function to the speech length ratio received from the speech length ratio calculating module 140, and then the decoded scores are mapped to the converted mapping scores. The process of calculating the intonation score may be performed by multiplying the weighted speech length ratio. The pronunciation score and the intonation score thus calculated may be transferred to the final similarity score calculation module 180 and may be converted into a final similarity score by reflecting a weight. The detailed operation of each detailed module will be described in detail with reference to the accompanying drawings.

The voice similarity determining apparatus according to the present invention can provide the learner with a desirable feedback effect by scoring and providing the similarity between the reference voice and the test voice.

3 is a diagram illustrating a situation in which the voice similarity determining apparatus according to the present invention may be modified.

As described above, the voice similarity determining apparatus according to the present invention may receive a reference voice signal and a test voice signal by integrating them in one device, and determine and score similarities between the two voice signals. Similarity between the two voice signals may be determined by cooperation between the terminal device 10, the external server 20, and the reference voice database 190. That is, a heavy load, such as preprocessing or determining similarity of the reference voice signal included in a video file such as a movie, advertisement, drama, etc., is performed by the external server 20 to receive a test voice signal or determine similarity. The differentiation of the work can be done so that only through the terminal device 10 such as a smart phone to provide results. An embodiment in which the apparatus for determining the voice similarity according to the present invention is divided into two parts as described above will be described in more detail with reference to the accompanying drawings.

4 is a view showing a modified embodiment of the voice similarity determining apparatus according to the present invention.

The apparatus for determining the voice similarity according to the present invention may be configured as one device as shown in FIG. 2, but the processing of the reference voice signal by dividing its function is performed by the external server 20, and the processing for the test voice signal. A part for determining similarity between the reference voice signal and the test voice signal may be dividedly provided to be performed by the terminal device 10. That is, the preprocessing process for the reference voice signal is all performed at the external server 20, and the terminal device 10 receives only the first voice length of the reference voice signal and the first MFCs and the similarity with the test voice signal. It can be provided by calculating the score.

Through such work differentiation, the voice similarity determining apparatus according to the present invention can achieve the effect of distributing and reducing the load generated during the preprocessing and the similarity determining process of the voice signal.

5 is a view showing another modified embodiment of the voice similarity determining apparatus according to the present invention.

Although the above-described embodiment has a load distribution, the load generated by the terminal device 10 is considerably higher than that of the external server 20. Typically, since the external server 20 is superior in terms of the calculation amount or capacity than the terminal device 10 such as a smart phone, it may be more preferable to reverse the role. That is, only the preprocessing part for the test voice signal is performed by the terminal device 10, and only the second voice length and the second MFCs are transmitted to the external server 20, and the external server 20 transmits the reference voice signal. It may be more desirable to determine similarity between test speech signals. Through such modified embodiments, the voice similarity determining apparatus according to the present invention may achieve an effect of significantly reducing the amount of work to be performed in each of the plurality of terminal devices 10.

6 is a view showing another modified embodiment of the voice similarity determining apparatus according to the present invention.

In the two modified embodiments described above, a preprocessing process is performed in which a noise and / or a background sound are removed from a reference voice signal and / or a test voice signal, and the voice length and the silent period are recognized and extracted to extract the voice length and the MC. do. However, in the case of the reference voice signal, the first voice length and the first MFC of the reference voice signal may be extracted and stored in a database by performing a preprocessing process in advance. That is, when the second voice length and the second MFC of the test voice signal are received by the terminal device 10, the first voice length and the first MFC extracted from the reference voice signal included in a movie, advertisement, drama, or the like. The first voice length and the first MFCs are called from the reference voice database 190 storing the seeds and transferred to the speech length ratio calculation module 140, the decodeable U module 150, and the like. Similarity between signals can be calculated and scored faster

7 is a block diagram illustrating an example of a filter module of an apparatus for determining voice similarity according to the present invention.

As described above, the first filter module 111 for removing noise and / or background sound from the reference voice signal and the second filter module 112 for removing noise and / or background sound from the test voice signal provide convenience of discrimination. To this end, only the first 'first' and 'second' modifiers are attached, and their roles or configurations may be the same. Herein, the filter module will be uniformly described. The same applies to the first voice interval detection module 121, the second voice interval detection module 122, the first FM extracting module 131, and the second FM extracting module 132.

The filter module includes noise spectrum section 110-2, fast fourier transform (FFT) section 110-1, winner filter section 110-3, and IFFT. (IFFT, Inverse Fast Fourier Transform) section 110-4, and through this configuration, it is possible to remove noise and / or background sound included in the voice signal. More specifically, the noise spectrum requesting section 110-2 estimates the noise spectrum and transmits the noise spectrum to the winner filter section 110-3 in the absence of speech, and receives the winner filter section 110-3. Section 110-1 may perform the operation of removing noise and / or background sound from the speech signal by minimizing the difference between the speech signal transmitted through the Fourier transform and the filtered signal. In addition, the speech signal from which noise is removed through the Wiener filter section 110-3 may be inversely transformed through the IFF section 110-4 and transferred to the speech section detection module.

This process can be summarized as follows.

Voice signals are a mixture of noise and clear signals.

 Where s [n] is a clean signal and z [n] represents a noise signal.

If the voice signal is output through a filter,

It can be represented as

이 깨끗한 신호 s[n]과 MSE(Mean Square Error)가 최소가 되도록 하도록 조정하며, 이는 아래 식을 따라 이루어질 수 있다.Here, the Wiener filter section 110-3 is an estimated signal

This clean signal s [n] and MSE (Mean Square Error) are adjusted to be minimal, which can be done according to the following equation.

Then, the Wiener-hopf equation is derived as

The Fourier transform can be used to calculate the following equation.

이고,

가 되며, 최종적으로는 아래와 같은 수식을 얻을 수 있다.Where clean signal s [n] and noise signal z [n] are stochastic independent

ego,

Finally, the following equation can be obtained.

8 is a diagram illustrating voice waveforms and spectrograms before and after Wiener filtering.

First, (a) of FIG. 8 shows a voice signal waveform and spectrogram before passing through the Wiener filter section 110-3, and (b) shows a voice signal waveform passing through the Wiener filter section 110-3. This diagram shows the spectrogram. It can be confirmed through the drawing that components such as noise included in the voice signal are well removed.

The apparatus for determining the voice similarity according to the present invention removes noise and / or background noise from the reference voice signal and the test voice signal through this process, thereby making it easier to detect and characterize the voice section in the reference voice signal and the test voice signal. It provides the effect of making it happen.

9 is a diagram illustrating an example of a speech section detection module of the apparatus for determining speech similarity according to the present invention.

The apparatus for determining speech similarity of the present invention may perform a process of detecting a speech section from a speech signal as a preprocessing operation for reducing the alignment error rate of the decodeable oil and calculating the similarity similarly. It may be made through the interval detection module 121 and the second voice interval detection module 122.

The detection of the negative intervals includes energy, zero crossing rate, autocorrelation coefficient, increase and decrease coefficient, linear predictive coding (lpc) coefficient, prediction error, differential energy, snr (siganl to noise ratio). It can be detected using any one or more. To this end, the voice interval detection module includes a frame unit processing section 120-1, a third hamming window section 120-2, a frame unit energy calculation section 120-3, and a zero crossing rate calculation section 120-. 4), the voice section determination section 120-5. More specifically, the speech section detection module processes the speech signal from which the noise and / or the background sound is removed through the filter module in the frame unit through the frame unit processing section 120-1 to perform the third hamming window section 120-2. , The energy calculation branch consisting of the frame energy calculation section 120-3 and the zero crossing rate calculation branch consisting of the zero crossing rate calculation section 120-4, and the third voice section determination section 120-5 The energy section and the zero crossing section of the received speech signal may be detected to recognize and recognize the speech section and the silent section of the speech signal.

Through this, the apparatus for determining speech similarity according to the present invention may achieve an effect of reducing the alignment error rate in the subsequent speech similarity determination.

10 is a view showing another example of the voice interval detection module of the voice similarity determination apparatus according to the present invention.

In addition, the voice interval detection module of the voice similarity determination apparatus according to the present invention includes a filter bank section 120-6, a band energy estimation section 120-7, and a voice interval detection section 120-8. A background noise adaptation section 120-9, a band SNR summation section 120-10, a threshold adaptation section 120-11, and a fourth speech segment determination section 120-12. May be This configuration is for implementing a standard Voice Activity Detection (VAD) standard of the European Telecommunication Standards Institude (ETSI). After dividing the input voice signal into nine subbands, The speech section may be determined by calculating subband energy using an SNR estimate.

11 is a block diagram showing the internal configuration of the MFC extraction module of the voice similarity determination apparatus according to the present invention.

The apparatus for determining speech similarity according to the present invention extracts MFCs from each of a reference speech signal and a test speech signal and determines the similarity by using the de-doubled oil. For this purpose, the MFC extraction module includes a preemphasis. Section 130-1, fourth Hamming window section 130-2, Discrete Fourier Transform (DFT) section 130-3, mel scale filter bank section 130 -4), log section 130-5, and discrete cosine transform (DCT) section 130-6.

A brief review of the role of each subsection of the MCC extraction module is as follows.

는 i번째 엠에프씨씨 차수의 평균 값, j는 프레임 번호를 의미한다.The pre-emphasis section 130-1 of the MFC extraction module may increase the energy of the high frequency region to increase the accuracy of phoneme detection, and the fourth hamming window section 130-2 is extracted from the speech. It makes the value of the signal at the boundary of the frame frame close to 0 and discretizes, and the DFT section 130-3 is divided from the windowed frame transmitted through the fourth Hamming window section 130-2. spectrum) can extract information. That is, it may serve to extract energy in each frequency band from the hamming windowed frame. Next, each frequency band energy extracted through the DFT section 130-3 is filtered through the Mel scale filter bank section 130-4 and the log section 130-5 modeled similarly to the human ear, These results are obtained through coefficient sections 130-6 to obtain coefficient values that provide useful features for finding phonemes. Particularly, in the present invention, 13th MFCs are used, and the average of each order of the 13th MFCs is the same over the entire ignition and CN (Cepstral Mean Normalization) is applied for energy leveling. The formula for this is as follows, i is the order of MFC,

Is the average value of the i-th order of MC, and j is the frame number.

The apparatus for determining the voice similarity according to the present invention can achieve the effect of extracting the characteristics of the reference voice signal and the test voice signal through the MC module.

12 is a diagram illustrating a process of extracting a feature from a voice signal by moving a frame.

Since the voice is an abnormal signal, the feature is extracted assuming that the voice signal is static in a small window as shown in the figure without extracting the feature throughout the spoken voice signal. The voice signal extracted from the window is defined as a frame.

13 is a view showing the shape of the hamming window.

The frames extracted from the speech are multiplied by the Hamming window, which makes the value of the speech signal at the boundary of the window close to zero, which is the fourth Hamming window section 130-2 of the MC extraction module. It can be done through. Then, the Hamming window is expressed as an equation as follows.

FIG. 14 is a view showing a triangular filter bank provided through a mel scale filter bank section.

The mel scale filter bank section 130-4 of the MC extract module has a constant interval of center frequency and a bandwidth of 100 Hz below 1 kHz, and above 1 kHz the center frequency and bandwidth increase rapidly in proportion to the log. It is a section having a filter bank, and the mel scale frequency in the frequency domain may be defined as follows.

The filter bank has various methods such as an angle, a square, a ladder, a Gaussian, and the mel scale filter bank section 130-4 used in the present invention uses a triangular weight as shown below. It is implemented by summing the calculated values.

는 m번째 삼각 필터 뱅크를 나타내고, f[m]은 중심 주파수를 나타내는데, 본 발명에 따른 음성 유사도 판단 장치는 이러한 멜 스케일 필터 뱅크 섹션(130-4)을 이용하여 수신되는 음성 신호에서의 특징을 사람의 귀의 특성과 유사한 방식을 이용하여 추출할 수 있다.In the above expression

Denotes the m-th triangular filter bank, and f [m] denotes the center frequency, and the voice similarity determining apparatus according to the present invention uses the mel scale filter bank section 130-4 to characterize the received voice signal. Can be extracted using a method similar to that of a human ear.

FIG. 15 is a diagram illustrating a process of calculating detired double scores made through the detired double module of the voice similarity determining apparatus of the present invention.

The DTI double-use module 150 of the voice similarity determining apparatus according to the present invention includes first and second MCs from the first MC extracting module 131 and the second MC extracting module 132, respectively. Seeds can be passed to calculate detiable double scores, which can be done by finding the optimal path length that minimizes warping cost. More specifically, the detired oil is an algorithm for finding a path that minimizes the distance between two data having different lengths. The length of the detired oil path may have an inequality range as shown in the following equation.

The DTW (R, T) that minimizes the warping cost among many warping passes is as follows.

In addition, the equation for applying dynamic programming to find the optimal path is as follows.

The optimal paths are obtained from the deddable distance distance matrix using the obtained deddable points. More specifically, after the deddable points on the optimal path are added, the detired level is calculated by leveling the length of the optimal path. Will be calculated.

The apparatus for determining voice similarity according to the present invention can effectively grasp the similarity between the reference voice signal and the test voice signal having different lengths through the decoded u module 150.

FIG. 16 is a view illustrating a process of converting a decoded U score into a mapping score through a pronunciation score calculation module and / or an accent score calculation module of the present invention speech similarity determination device.

Since the smaller the detiable double score means that the similarity between the reference voice signal and the test speech signal is higher, it is necessary to convert the detired double score into the detired double score in units of 100 points that are familiar to the user. This can be done through the following formula, in which the minimum value of the deity doubled points is the maximum score and the maximum value of the deity doubled scores is the minimum score.

Here, X denotes a detitable oil score, and the maximum score of the deity doubled oil (DTW) and the minimum score of debitable oil may be obtained from the entire data used in the implementation of the present invention as a predetermined score.

In addition, among the values calculated through the above equation, a conversion score larger than 100 is calculated as 100 points, a score smaller than 0 is output as 0, and the converted value may be used as a pronunciation score.

The apparatus for determining speech similarity according to the present invention may objectively evaluate how similar the pronunciation between the test speech signal spoken by the user and the reference speech signal is provided to the user through the pronunciation score calculation module 160. In addition, the simple evaluation technique of the pass and the non-pass cannot provide information on whether or not the pronunciation is improved. The apparatus for determining the voice similarity according to the present invention scores an evaluation of the pronunciation to provide a user's pronunciation. It can also be used as objective evidence of whether it is improving.

17 is a view showing the weight function used in the intonation score calculation module of the present invention speech similarity determination apparatus.

The intonation score calculation module 170 of the apparatus for determining speech similarity according to the present invention calculates the weighted speech length ratio by applying a weighting function to the speech length ratio provided by the speech length ratio calculating module 140 and multiplies it by the pronunciation score for the intonation. To calculate the score, Figure 17 is a diagram showing an example of the weight function used in this process. More specifically, the function of the drawing has a weight of 1 when the speech length ratio is 1, and has a weight of 0 when the speech length ratio is 0.5 or less and 1.5 or more. In addition, when the ignition length ratio is 0.5 or more and 1 or less, it has a slope value of 2, and when the ignition length ratio is 1 or more and 1.5 or less, it has a slope value of -2. However, this is only one example and various weight functions may be applied as necessary.

The apparatus for determining speech similarity according to the present invention has an effect of scoring and providing not only the pronunciation characteristics of the user but also the intonation characteristics through the intonation score calculation module 170.

18 is a flowchart illustrating a processing procedure of a voice similarity determining method according to the present invention.

First, in the voice similarity determining method according to the present invention, noise and / or background sound are removed from the reference voice signal and / or the test voice signal (S18-1), and the first voice is removed from the reference voice signal from which the noise and / or background sound is removed. After calculating the voice length (S18-2), a process of extracting the first MFCs may be performed. (S18-3) A similar process is performed on the test voice signal, and the second voice on the test voice signal. After the length is calculated (S18-4), a process of extracting the second MFCs may be performed. (S18-5)

After the first voice length and the first MFCs are extracted from the reference voice signal and the second voice length and the second MFCs are extracted from the test voice signal, the first voice length is divided by the second voice length. The process of calculating the length ratio is performed (S18-6), the D-T double scores are calculated between the first MFCs and the second M-Cs (S18-7), by mapping the calculated D-T double points A process of converting the mapping score to the mapping score may be performed. (S18-8) Next, a weight function is applied to the speech length ratio to calculate the weighted speech length ratio (S18-9), and the weighted speech length ratio is mapped to the mapping score. The process of multiplying the accent score may be performed (S18-10).

When the calculation of the accent score and the pronunciation score is completed, the process of calculating the final similarity score by adding the two values is performed. The two scores are not arithmetically added, but the pronunciation score is calculated using the mapping score (S18-11). The process of calculating the weighted pronunciation score by multiplying the pronunciation weight may be performed. (S18-12) In addition, the accent score is multiplied by the accent weight to calculate the weighted accent score (S18-13). The weight accent score is added to calculate and provide the final similarity score. (S18-14)

The inventors' voice similarity determination method can achieve the effect of improving the learning efficiency by providing appropriate feedback to the learner or the user through providing the final similarity score calculated through this process.

19 is a flowchart illustrating a modified embodiment of the voice similarity determining method according to the present invention.

As described above, the apparatus for determining voice similarity according to the present invention may be configured by dividing a role into an external server 20 and a terminal device 10, and FIG. 19 illustrates a first voice length of a reference voice signal from an external server 20. And a terminal device receiving the first MFCs may determine the similarity between the reference voice signal and the test voice signal.

The method of determining voice similarity according to this modified embodiment may be first performed from the process of receiving the first voice length and the first MFC of the reference voice signal from the external server 20 or the like. (S19-1) In parallel, a process of removing noise and / or a background sound of a test voice signal (S19-2), a process of calculating a second voice length from the test voice signal (S19-3) and a process of extracting the second MCs This may be performed simultaneously or concurrently. (S19-4)

After the extraction of the second voice length and the second MFC of the test voice signal is completed, the above-described process may be similarly performed in the terminal device 10, and more specifically, the first voice length may be determined as the second voice length. The speech length ratio is calculated by dividing by the voice length (S19-5), the DTI double scores between the first MCs and the second MSs are calculated (S19-6), and the DTI double scores are mapped and mapped. A process of converting the score to the score may be performed (S19-7). The weighted speech length ratio is calculated by applying a weight function to the speech length ratio (S19-8), and the weighted speech length ratio is applied to the mapping score. The process of calculating the intonation score (S19-9) and the fixing to calculate the pronunciation score using the mapping score may be performed. (S19-10)

After the pronunciation score and the accent score are calculated, the weighted pronunciation score and the weighted pronunciation score may be calculated by applying weights to the pronunciation score and the intonation score (S19-11) and the final similarity. The score may be calculated and provided through the sum of the weighted pronunciation score and the weighted pronunciation score. (S19-13) Here, the pronunciation weight may be 0.52 and the accent weight may be 0.48.

The voice similarity determination method according to the present invention has the effect of distributing and reducing the load generated during the preprocessing and the similarity determination process of the voice signal through such task differentiation.

20 is a flowchart illustrating another embodiment of a method for determining speech similarity according to the present invention.

FIG. 20 is a process in which the external server 20 receives the second voice length and the second MFCs transmitted by the terminal device 10 to determine the similarity between the reference voice signal and the test voice signal. Is a flow chart showing.

To this end, the voice similarity determination method according to another embodiment receives the second voice length and the second MFCs of the test voice signal from the terminal device 10 (S20-1), and the noise and / or in the reference voice signal After removing the background sound (S20-2), a process of calculating the first voice length from the reference voice signal may be performed. (S20-3) In addition, a process of extracting the first MFCs may be performed. (S20-4) After the calculation of the first voice length and the extraction of the first MFCs are completed, the speech length ratio is calculated by dividing the first voice length by the second voice length (S20-5) and the first M The process of calculating the decoded U scores calculated between the FCs and the second MCs may be performed (S20-6), and the process of converting the scores to the mapping scores may be performed (S20-7). The ratio is calculated (S20-8), and the weighted speech length ratio is applied by applying the mapping score. The accent marks may be calculated and the process (S20-9), utilizing the mapping score of pronunciation score calculation can be performed. (S20-10)

Finally, a weighted pronunciation score is calculated by applying a pronunciation weight of 0.52 to the pronunciation score (S20-11), and an accent weight score is calculated by applying an accent weight of 0.48 to the accent score (S20-12). The process of calculating the final similarity score by adding the two values may be performed. (S20-13)

Through such an embodiment, since a complicated and computational process is performed in the external server 20, the workload on the terminal device 10 may be reduced.

FIG. 21 is a flowchart illustrating still another embodiment of the present invention.

The test voice signal should be processed at that time, whereas the reference voice signal is an audio signal included in an image or the like, and thus the first voice length and the first voice length may be previously processed through a preprocessing process such as noise and / or background sound. Extraction of the first MFCs is possible. It is also possible to store such data in a database in advance and call it at that time to use it for similarity judgment. FIG. 20 illustrates this embodiment, and according to another embodiment, a method of determining a voice similarity receives a second voice length and a second MFC from the terminal device 10 (S21-1), and a reference voice database. After receiving and calling the first voice length and the first MFCs from (S21-2), the speech length ratio can be calculated by dividing the first voice length by the second voice length (S21-3). The conversion to the mapping score is performed by using the T-Double oil score (S21-4) calculated between the 1 CM and the second MC (S21-5), and by applying a weight function to the speech length ratio A process of calculating the accent score may be performed by applying the calculated weighted speech length ratio (S21-6) to the mapping score. (S21-7) Then, the pronunciation score is calculated using the mapping score (S21-). 8) After the pronunciation score and accent score have been calculated, The weighted pronunciation score and the weight accent score are calculated by applying the median (S21-9), and the process of calculating the final similarity score by summing these two values can be completed (S21-11).

According to the voice similarity determination method according to another embodiment of the present invention, since the first voice length and the first MFC of the reference voice signal are stored in a database in advance, the similarity determination calculation and provision on both signals are There is an effect that can be scored faster and more effectively.

22 is a flowchart illustrating a process of transmitting the second voice length and the second MFCs from the terminal device configuring the voice similarity determination device according to the present invention to an external server.

The apparatus for determining voice similarity according to the present invention may include an external server 20 and a terminal device 10 in which roles are shared. In this case, the terminal device 10 may calculate the second voice length from the test voice signal spoken by the user (S22-1) and then extract the second MFCs (S22-2). The extracted second MFCs and the second voice length may be transmitted to the external server 20 (S22-3), and the external server 20 receiving the second MFCs and the second voice length may be transmitted. By using the length, the similarity between the test voice signal and the reference voice signal may be scored and determined, and the result may be transmitted back to the terminal device 10.

FIG. 23 is a table showing correlations between score results calculated according to a voice similarity determining apparatus and a voice similarity determining method according to the present invention and score results evaluated by expert scorers.

First, the specifications of the reference voice and the test voice used for the performance experiment of the present invention are as follows.

The performance test was conducted using a wave file consisting of a test voice signal consisting of 28 reference voice signals and 100 test voice signals. In this case, the reference voice signal is a voice signal obtained by extracting the actor's speech in a length of 1 to 4 seconds in a movie or drama in which noise and background music exist, and the test voice signal is similar to a real user according to the reference voice signal. Voice signals were used to imitate, deliberately pronounce other words, or be fluent.

As can be seen through the drawings, the correlation coefficient between the score calculated by the development module of the voice similarity determining device and the voice similarity determining method according to the present invention and the average value of the scores evaluated by the scorers is 0.3823, It can be seen that the result calculated according to the present invention is a value providing a reasonable criterion in that there is no significant difference from 0.3795, which is a correlation coefficient between three scorers having a high correlation coefficient.

FIG. 24 is a table showing correlations between scores calculated by using a pronunciation score and an accent score calculated through the present invention in a linear regression model and score results evaluated by expert scorers.

The correct answer of the linear regression model was the average score of the scorers. The total data were divided into 10 groups of 10 by 10, and 9 groups were used for training and 1 group was used for testing. The correlation coefficient between the conversion scores for the 100 data predicted through the regression model and the average scores among the professional scorers was calculated. The above process was repeated 100 times, and after obtaining the correlation coefficient value between the development module and the grader according to the present invention, the average of 100 correlation coefficient values was used as the correlation coefficient value between the development module and the grader according to the present invention. Each time this process is repeated, 10 groups are randomly grouped. This can be expressed as follows.

Checking the values summarized through the drawings, the correlation coefficient value of the similarity score and the average of the scorers calculated by applying the development module according to the present invention to the linear regression model is 0.3313, the scorer having the highest correlation coefficient with the development module It can be seen that the correlation coefficient between the three persons is 0.3457, and as a result, the result calculated by directly applying the voice similarity determining device and the voice similarity determining method according to the present invention is a scorer rather than learning by applying a linear regression model. The higher correlation with, the higher the reliability.

As described above, although described with reference to preferred embodiments of the present invention, those skilled in the art will be variously modified without departing from the spirit and scope of the invention described in the claims below. And can be changed.

10: terminal device
20: external server
110-1: FT section
110-2: Noise Spectrum Hearing Section
110-3: Winner Filter Section
110-4: IFF Section
111: first filter module
112: second filter module
120-1: Frame Unit Processing Section
120-2: third hamming window section
120-3: Frame Energy Calculation Section
120-4: Zero crossing rate calculation section
120-5: Third voice section discrimination section
120-6: Filter Bank Section
120-7: Band Energy Estimation Section
120-8: Voice section detection section
120-9: Background Noise Adaptation Section
120-10: Band SN Combined Section
120-11: Threshold adaptation section
120-12: fourth speech section discrimination section
121: first voice section detection module
122: second voice interval detection module
130-1: pre-emphasis section
130-2: fourth hamming window section
130-3: FT section
130-4: Mel scale filter bank section
130-5: log section
130-6: DC section
131: first MC extract module
132: second MFC extraction module
140: ignition length ratio calculation module
150: D Twice module
160: pronunciation score calculation module
170: intonation score calculation module
180: final similarity score calculation module
190: reference speech database

Claims (30)

Calculating a first speech length from the reference speech signal;
Extracting first mel frequency cepstal coefficients (mfcc) from the reference speech signal;
Calculating a second speech length in a test speech signal;
Extracting second MFCs from the test voice signal;
Calculating a speech length ratio by dividing the first voice length by the second voice length;
Calculating dynamic time warping (dtw) scores between the first MFCs and the second MFCs;
Mapping the decoded unique scores into a mapping score;
Calculating a weighted speech length ratio by applying a weight function to the speech length ratio;
Calculating the accent score by multiplying the mapping score by the weighted speech length ratio,
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination method, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
Receiving a first voice length and a first MFC of a reference voice signal from an external server;
Calculating a second speech length in the test speech signal;
Extracting second MFCs from the test voice signal;
Calculating a speech length ratio by dividing the first voice length by the second voice length;
Calculating detired double scores between the first MFCs and the second MFCs;
Converting the decodeable scores into a mapping score;
Calculating a weighted speech length ratio by applying a weight function to the speech length ratio;
Calculating the accent score by multiplying the mapping score by the weighted speech length ratio,
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination method, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
Receiving a second voice length and second FMs of the test voice signal from the terminal device;
Calculating a first speech length from the reference speech signal;
Extracting first MFCs from the reference voice signal;
Calculating a speech length ratio by dividing the first voice length by the second voice length;
Calculating detired double scores between the first MFCs and the second MFCs;
Converting the decodeable scores into a mapping score;
Calculating a weighted speech length ratio by applying a weight function to the speech length ratio;
Calculating the accent score by multiplying the mapping score by the weighted speech length ratio,
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination method, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
Receiving a second voice length and second FMs of the test voice signal from the terminal device;
Receiving a first voice length and a first MFC from a reference voice database;
Calculating a speech length ratio by dividing the first voice length by the second voice length;
Calculating detired double scores between the first MFCs and the second MFCs;
Converting the decodeable scores into mapping scores through mapping;
Calculating a weighted speech length ratio by applying a weight function to the speech length ratio;
Calculating the accent score by multiplying the mapping score by the weighted speech length ratio,
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination method, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
The method according to any one of claims 1 to 4,
The weight function is
When the ignition length ratio is 1 has a weight value of 1,
The speech similarity determining method has a weight value of 0 when the speech length ratio is 0.5 or less and 1.5 or more.
The method of claim 5,
The weight function is
In the section of the ignition length ratio of 0.5 or more 1 has a slope value of 2,
The speech similarity determination method of claim 1, wherein the speech length ratio has a slope value of -2 in a section of 1 to 1.5.
delete delete The method according to any one of claims 1 to 4,
In the step of calculating the detired double scores between the first MFC and the second MFC,
The debit double points
The method of claim 1, wherein the paths are obtained by adding values on a path in a demultiplexing distance matrix created by using the first and second MCs.
The method according to any one of claims 1 to 4,
The step of converting the deddable U scores are converted into mapping scores
The minimum value of the decoded double points is the maximum score,
The minimum value of the maximum number of deddable existence scores

Are converted to mapping scores using equations,
The detired oil score is obtained by equalizing the length of the optimal path by adding the value of the optimal path in the detired oil distance matrix,
And a maximum score of the detiable double and a minimum score of the delimited double are predetermined.
The method of claim 1,
Before the step of calculating the first speech length in the reference speech signal,
And removing the noise and / or the background sound from the reference voice signal and / or the test voice signal.
The method of claim 2,
Prior to the step of calculating the second speech length in the test speech signal,
And removing the noise and / or the background sound from the test voice signal.
The method of claim 3,
Before the step of calculating the first speech length in the reference speech signal,
And removing noise and / or background sound from the reference voice signal.
The method according to any one of claims 11 to 13,
Noise and / or background noise is removed through a winner filter.
delete delete A first speech interval detection module for calculating a first speech length from a reference speech signal;
A second voice interval detection module for calculating a second voice length from the test voice signal;
The first voice length is received from the first voice interval detection module, the second voice length is received from the second voice interval detection module, and the first voice length is divided by the second voice length to speak length ratio. Ignition length ratio calculation module for calculating;
A first MC extracting module for extracting first MCs from the reference speech signal;
A second MC extracting module for extracting second MCs from the test voice signal;
A decoded-deowable module receiving the first MCs from the first MC extracting module and receiving the second MCs from the second MC extracting module to calculate detired oil scores;
The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio;
A pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping;
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination device, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
A storage module for receiving and storing the first voice length and the first MC of the reference voice signal from an external server;
A second voice interval detection module for calculating a second voice length from the test voice signal;
An utterance length for receiving the second voice length from the second voice interval detection module and receiving a first voice length from the storage module and dividing the first voice length by the second voice length to calculate a utterance length ratio Non-calculating module;
A second MC extracting module for extracting second MCs from the test voice signal;
A DTI double U module receiving first MFCs from the storage module, and receiving second MFCs from the second MFC module to calculate detired double scores;
The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio;
And a pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping.
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination device, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
A storage module for receiving and storing the second voice length and the second FM of the test voice from the terminal device;
A first voice interval detection module configured to calculate a first voice length from a reference voice signal;
A utterance length for receiving a first voice length from the first voice interval detection module and receiving a second voice length from the storage module and dividing the first voice length by the second voice length to calculate a speech length ratio. Non-calculating module;
A first MC extracting module for extracting first MCs from the reference speech signal;
A DTI double U module receiving second MFCs from the storage module, and receiving first MFCs from the first MFC module to calculate detired double scores;
The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio;
A pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping;
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination device, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
A storage module for receiving and storing the second voice length and the second FM of the test voice from the terminal device;
A reference voice database storing the first voice length of the reference voice and the first MFCs;
Speech length ratio calculation for receiving the first speech length from the reference speech database, receiving the second speech length from the storage module, and calculating the speech length ratio by dividing the first speech length by the second speech length. module;
A deddable-you module for receiving the first MFCs from the reference voice database, receiving the second MFCs from the storage module, and calculating a decoded-U scores;
The weighted speech length ratio is calculated by applying a weight function to the speech length ratio received from the speech length ratio calculating module, and the weighted speech is applied to the mapping score converted from mapping by receiving the DTI double-U scores from the DTI-D module. An accent score calculation module that calculates an accent score by multiplying the length ratio;
A pronunciation score calculation module configured to receive the decoded U scores from the decoded U module and calculate a pronunciation score using the mapping score converted through the mapping;
The first MFCs and / or the second MFCs may be pre-emphasis, hamming window, and discrete fourier transform (DFT) sequentially. Obtained by applying a melt scale filter bank and a Discrete Cosine Transform (DCT),
The voice similarity determination device, characterized in that extracted by applying the CMN (Cepstral Mean Normalization) for the energy level up to the 13th order.
The method according to any one of claims 17 to 20,
The weight function is
When the ignition length ratio is 1 has a weight value of 1,
The speech similarity determining device having a weight value of 0 when the speech length ratio is 0.5 or less and 1.5 or more.
The method of claim 21,
The weight function is
In the section of the ignition length ratio of 0.5 or more 1 has a slope value of 2,
The speech similarity determination device, characterized in that having a slope value of -2 in the section of the speech length ratio of 1 to 1.5.
The method according to any one of claims 17 to 20,
Calculate a weighted pronunciation score by multiplying the pronunciation score received from the pronunciation score calculation module by multiplying the pronunciation weight, multiplying the accent score received from the accent score calculation module by the accent weight to calculate a weighted accent score, and then the weighted pronunciation And a final similarity score calculation module for adding a score and the weight accent score to calculate a final similarity score.
The method of claim 23,
The pronunciation weight is 0.52,
And the accent weight is 0.48.
The method according to any one of claims 17 to 20,
The deddable module
The apparatus of claim 1, wherein the values of paths are calculated by adding values on a path in a demultiplexing distance matrix generated by using the first and second MCs.
The method according to any one of claims 17 to 20,
The mapping score used by the intonation score calculation module and / or the pronunciation score calculation module is
Received from the Detty Double U module
The minimum value of the decoded double points is the maximum score,
The minimum value of the maximum number of deddable existence scores

Is converted to the mapping score using an equation,
The detired oil score is obtained by equalizing the length of the optimal path by adding the value of the optimal path in the detired oil distance matrix,
The detired double maximum score and the detired double minimum score are predetermined.
The method of claim 17,
A first filter module which removes noise and / or background sound from the reference voice signal and transmits the noise to the first voice interval detection module;
And a second filter module which removes noise and / or background sound from the test voice signal and delivers the noise to the second voice section detection module.
The method of claim 18,
And a second filter module which removes noise and / or background sound from the test voice signal and delivers the noise to the second voice section detection module.
The method of claim 19,
And a second filter module which removes noise and / or background sound from the reference voice signal and transmits the noise and / or background sound to the second voice interval detection module.
delete

Images