JP2002040926A - Foreign language-pronunciationtion learning and oral testing method using automatic pronunciation comparing method on internet - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a language learning method capable of comparing the voice of a native speaker with that of a leaner quickly and correctly through the automatic voice comparing algorithm of a dynamic time warping base without recognizing the voice of the learner. SOLUTION: This method with which a learner can improve listening comprehension and practice conversation of a foreign language by using a computer connected to the Internet and also which is capable of objectively testing the conversational ability of the leaner is provided with a learner's voice signal 10, a native speaker's voice signal 20, an automatic voice comparing network 30, a DTW (dynamic time warping) base difference comparing model 40, an error correcting nerve circuitry 50, an expert evaluation comparing network 60 and an error calculating network 70.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for providing a learning service for foreign language pronunciation learning and an oral test using an automatic first-book comparison method on the Internet. In particular, native speaker audio, learner audio,
A test to learn the automatic speech comparison network using data consisting of expert evaluations, compare the pronunciation of the learner with the pronunciation of the native speaker using the learned automatic speech comparison network, and determine the accuracy of the pronunciation. It relates to a service for interactive bilingual pronunciation learning and oral testing that can be performed.

[0002]

2. Description of the Related Art In a conventional method of learning a foreign language, listening and conversation learning are performed by repeating the pronunciation of a native speaker through a cassette tape or a videotape, and the learner imitates the pronunciation and resembles the pronunciation of a native speaker. While learning the degree, they learned repeatedly so that their pronunciation was accurate.

[0003] Since such a learning method cannot objectively evaluate the pronunciation of a foreign language, efforts have been made to find the difference in pronunciation between oneself and a native speaker through an objective scale.

[0004] That is, conventionally, the sound difference in the time domain,
For example, a method has been mainly used in which the difference between the ton of the voice signal and the overall pronunciation time is simply compared, and the pronunciation of the native speaker and the pronunciation of the learner are compared.

Recently, a pronunciation comparison method using audio signal processing technology has been developed, and the method is based on a hidden Markov model.
(Hidden Markov Model; hereinafter, referred to as HMM). Most of the algorithms compare the pronunciation of the learner's pronunciation with the native speaker's speech.

However, if the learner makes a pronunciation in an environment with ambient noise or the learner's pronunciation is indistinguishable and an error occurs in recognition, the difference from the native speaker pronunciation may become meaningless. Many.

In order to evaluate a learner's ability to listen and speak in a foreign language, a TSE (Test of Speakin
g English), SEPT (Spoken English Proficiency Te)
The student's foreign language proficiency could be evaluated by an interview system in which a professional evaluation test such as st) was directly asked and answered by a native speaker language expert at a designated time and place.

[0008]

However, these methods are still limited in time and space in testing foreign language proficiency, and the evaluation of experts is influenced by subjective factors such as the degree of fatigue and surrounding conditions. There was a problem that it was easy.

The present invention has been made in view of the above problems, and an object of the present invention is to provide dynamic time warping (Dynam warping).
ic Time Warping; ) Does not recognize the learner's speech through the underlying automatic speech comparison algorithm,
To provide a foreign language pronunciation and learning and oral test method using an automatic pronunciation comparison method on the Internet, which implements a language learning method capable of quickly and accurately comparing the difference with a native speaker voice. is there.

[0010] Another object of the present invention is to enable a learner to practice his / her foreign language pronunciation and take an oral test at a desired time and place regardless of time and place in a web-based state on the Internet. Another object of the present invention is to provide a foreign language pronunciation learning and oral test method using an automatic pronunciation comparison method on the Internet.

[0011]

SUMMARY OF THE INVENTION In order to achieve the above-mentioned object, the present invention relates to a method for learning foreign language pronunciation which automatically compares a native speaker's voice with a learner's voice and obtains the difference. The difference value between the phoneme and the prosody of the native speaker voice is calculated by the DTW-based difference comparison network, and the difference between the difference value and the comparison value evaluated by the expert evaluation comparison network is calculated by the error calculation network so that the difference value becomes smaller. A foreign language pronunciation learning and oral test method using an automatic pronunciation comparison method on the Internet to learn a DTW based difference network is presented.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be specifically described below with reference to the drawings showing the embodiments. FIG. 1 is a block diagram showing an automatic speech comparison network algorithm for foreign language pronunciation learning and an oral test on the Internet according to the present invention.

First, in order to implement an automatic voice comparison network algorithm, given data is a pronunciation difference value obtained by a specialist comparing words and sentences pronounced by a native speaker and a learner. Usually, the pronunciation difference value takes five discrete values (1 to 5).
Here, the voice of the native speaker is assumed to be data (transcribed date) in which phonetic symbols are displayed according to time.

Referring to the automatic speech comparison network algorithm shown in FIG. 1, the learner speech signal 10, the native speaker speech signal 20, the automatic speech comparison network 30, the expert evaluation comparison network 60 and the error calculation network 70 It is configured.

At this time, the difference between the phoneme and the prosody of the learner's voice signal 10 and the native speaker's voice signal 20 is calculated by the network 30. The automatic voice comparison network 30 further includes a DTW-based difference comparison network 40 and an error correction neural network 50.

An error calculation network 70 obtains a difference between the value of the expert evaluation comparison network 60 and the comparison value obtained by the automatic voice comparison network 30, and the error correction neural network 50 operates so that the difference value becomes small. Learning takes place.

FIG. 2 is a partial detailed view of the DTW-based difference comparison model shown in FIG. The DTW shown in FIG.
To explain the base difference comparison model (or DTW base difference comparison model network) in more detail, the phonetic difference and the prosody difference of the learner's voice signal 10 and the native speaker's voice signal 20 are calculated by the DTW base difference comparison model 40.

First, in the phoneme difference calculation model 42, a strength difference comparison block 42a, a time difference comparison block 42b, and a frequency difference comparison block 42c are calculated. This block calculates the accuracy of learner pronunciation for each syllable and phoneme.

In the strength difference comparison block 42a, the speaker (spe
After eliminating all the characteristics of the audio signal by aker), a difference between the two audio signals is obtained. In other words, the linguistic messages (lin
guistic message).
The time difference comparison block 42b is a block for calculating the difference in pronunciation duration of sentences, words, syllables, phonemes, etc. The frequency difference comparison block 42c is a formant position difference between the voice pronounced by the learner and the voice of the native speaker. 2 shows a block for calculating.

The phoneme difference calculation model 44 indicates a block for determining whether or not the learner correctly pronounces between phonemes, between syllables, between words, and the like in the entire sentence. In the phoneme difference calculation model 44, a stress difference comparison block 44a, an intonation difference comparison block 44b, and a rhythm difference comparison block 44c are calculated.

At this time, the phoneme calculation model 44 generates a pitch contour (pitch contour) of the learner and the native speaker voice.
ur) obtained from the difference. That is, an existing pitch detection method is used to determine the pitch shape according to time in the voices of the learner and the native speaker. The stress difference comparison block 44a is a block for calculating a relative position difference between the maximum peak values in the pitch contour line. The intonation difference comparison block 44b is a block obtained from the difference between the inclinations of the two pitch contour lines at the end of the learner and the native speaker pronunciation. The rhythm difference comparison block 44c is a block that includes a peak appearing between adjacent words and syllables. Fig. 4 shows a block calculated from the relative position and size of the valley.

DTW-based difference comparison model 40 as described above
6 output values calculated by
It passes through the error correction neural network 50 before being compared with the expert evaluation value of zero. At this time, the error correction neural network 50
Is a network for calculating the difference between phonemes and prosody, that is, a network for nonlinearly assembling the six output values of the DTW-based difference comparison network 40 so that the pronunciation comparison value calculated automatically approaches the expert evaluation value. As a structure of the error correction neural network 50, a multi-layer perse proton model (Multi-Layer Perceptron) of two layers is applied.

Numerical values calculated by the algorithm of the automatic pronunciation comparison network and the expert evaluation comparison network 6
For the expert evaluation value of 0, the error is calculated by the error correction neural network 50 and the synapse (synaps) of the signal network is calculated.
e) We will learn weights. At this time, learning is based on the existing Mean Squared Error Function and the error back propagation learning algorithm (Error-Back Propagation al
gorithm).

FIG. 3 is a detailed flowchart for comparing the strength difference in the phoneme difference calculation model shown in FIG. Referring to the detailed flowchart of the strength difference comparison block shown in FIG. 3, the calculation of the strength difference between the native speaker and the learner's voice is implemented by the following algorithm. First, an end point is extracted S1 from the audio signal S100 of the native speaker.
02, the end point is also extracted S103 in the learner's voice signal S101, and the signal passes through the energy standardization block S104. Next, after removing the difference in output energy from the microphones, frame blocking S105 and S106 are performed and Fourier transforms S107 and S108 are performed. At this time,
The frame blocking S105, S106 is a portion that divides the audio signal entering the time series into tens of milliseconds (milli-seconds) and covers a Hamming window or a Hanning window.

Also, Fourier transforms S107 and S108
Converts a time domain signal into a frequency domain signal by Fourier transform of two audio signals for each frame. Next, after performing time-frequency dynamic warping S109 for removing the characteristics of the speaker through linear frequency conversion and DTW, Bark (B
ark) frequency warping in units of S110 and S111;
Intensity warping in loudness units S112
2, through the step S113.

At this time, time-frequency dynamic warping S1
09 is the effect of the speaker difference between the native speaker's voice and the learner's voice, that is, the duration of the pronunciation (duratio
n) A block for eliminating a difference in a frequency domain due to a time difference and a vocal tract length difference. The difference in pronunciation duration can be removed by DTW, and the difference due to vocal tract length can be removed by linear frequency conversion.

Frequency Warping S in Bark Units
Reference numerals 110 and S111 denote audio signals in Hertz Hz as Bark (psychoacoustic) frequency units.
k) This is the part that changes into units. The intensity warping in units of loudness (S112, S113) is a block that converts the energy of the spectrum output through the Fourier transforms S107, S108 into a loudness unit, which is a psychoacoustic intensity unit.

After the inverse Fourier transforms S114 and S115, cepstrum calculation blocks S116 and S116
Finally, a cepstrum feature vector is extracted by 117. That is, Fourier inverse transform S114, S115
Since the signal subjected to loudness unit strength warping takes a real value and is symmetric, the cosine transform (Cosine tran
sform).

The audio signal S100 of the native speaker
The cepstrum feature vector to be compared with the learner's speech signal S101 is used as the cepstrum calculation blocks S116 and S116.
Finally, it is extracted by 117. At this time, the method of the present invention described above uses the existing PLP (perceptual linear predi
ction) Similar to the feature extraction method, except that a time-frequency dynamic warping block S109, which performs a time-frequency dynamic warping feature extraction process for eliminating speaker characteristics of the audio signal, is newly added. There is.

Next, after calculating the distance for each frame, S1
18, converted to phoneme intensity difference unit 11
9. At this time, the frame-by-frame distance calculation block S118
This part calculates the distance difference between the native speaker and the feature vector of the learner's voice in frame units. In the distance calculation, a Euclidean distance is calculated, and a distance difference value is output for every frame to obtain a pronunciation difference value of two voices.

A frame-by-frame distance calculation block S11
1 to compare the pronunciation difference value obtained in 8 with the expert evaluation value
In the phoneme intensity difference calculation block S117, linear transformation or logistics (logis
tic) transform.

FIG. 4 is a partial detailed view of the time-frequency dynamic warping stage shown in FIG. Referring to FIG. 4, the Fourier-transformed native speaker sound 20
0 and the learner's speech 201 are warped in the time and frequency domains so that the distance of each of these cepstrum feature vectors is minimized. That is, the learner's speech 201 passes through the linear frequency warping network 202 to eliminate the difference in pronunciation time from the native speaker speech 200 in order to eliminate the difference due to vocal tract length, which is cited as the main cause of the speech signal difference between speakers. Pass through a non-linear dynamic time warping 203.

Then, the process enters the blocks 204 and 205 for extracting the cepstrum feature vector, calculates the cepstrum, and calculates the Euclidean distance between the cepstrum vectors in the frame-by-frame distance calculation block 206. At this time, linear frequency warping 202 and non-linear dynamic time warping 203 are performed so that this error is minimized.

On the other hand, the time difference comparison block 42b of FIG. 2 can be calculated using the warping degree by time using the feature vector of the learner and the native speaker calculated by the time-frequency dynamic warping of FIG. That is, the output value of the time difference comparison block 42b is a value obtained by calculating all phoneme-based sounding duration differences of the two voice signals arranged on the time axis and dividing the difference by the total number of phonemes.

Similarly, in FIG.
The block performs a linear transformation based on the frequency axis in a manner similar to the method obtained in the time difference comparison block 42b, and then performs a first formant F1, a second formant F2, a second formant F2 between a voice pronounced by a learner and a voice of a native speaker. It is calculated from the position difference of three formants F3 and the like.

FIGS. 5A to 5C are comparison graphs of the prosody difference calculation model shown in FIG. As shown in FIG. 5, the prosody difference between the two voice signals is calculated from the difference between the pitch contour lines. And the pitch contour line in unvoiced pronunciation is continued.

Referring to FIG. 5A, there is shown a graph showing pitch contours of a native speaker and a learner's sentence pronunciation with time. The stress difference comparison block 44a in FIG. 2 compares the wrong degree of the syllable or word in which the maximum peak appears in the pitch contour line with the stress difference of the two voices. That is, when calculating the time difference between the syllable pronounced by the learner with stress and the stressed syllable of the native speaker, the number of syllables in the difference section appears as the stress difference.

Referring to FIG. 5B, this is to explain the intonation difference comparison block 44b of FIG. 2, and the intonation difference is calculated from the inclination difference of the pitch contour line appearing at the end of the sentence pronunciation. That is, in the case of ordinary sentences, the pitch inclination at the end of the sentence is a negative number, and in the case of a question sentence, the inclination is generally a positive number. The difference between the intonations of the two voices can be obtained using the above-described inclination difference.

FIG. 5c is a graph illustrating the rhythm difference comparison block 44c of FIG. 2, showing a triangular peak at the pitch contour of two voices, and an inverted triangle representing a valley.
y). The rhythm difference between the two voices can be obtained from the difference between the number and the size of the peak and the barry.

FIG. 6 is a schematic block diagram of an Internet system for learning a foreign language pronunciation and an oral test according to the present invention. As shown in FIG. 6, the server computer 300 for foreign language pronunciation learning and an oral test includes:
Internet 320 connection network connected to the server computer 30, learner computers 340a, 340 connected via the Internet 320 connection network
, and learners 360a, 360b,... At this time, learners 360a, 360b, ...
Has a microphone device and a speaker so that it can listen to and record audio signals.

FIG. 7 shows a series of data processing steps performed between the computer and the server computer. FIG. 7a shows that most of the algorithms are processed on the learner computer, and FIG. 7b shows that most of the algorithms are processed on the server computer. First, FIG.
To explain a, the learner computer 340 uses a learner's voice signal 342 obtained by recording the learner's voice pronunciation via a microphone, and a native speaker voice from the native speaker voice signal database 302 of the server computer 300 to practice pronunciation. The signal 304 is taken, the difference between the two voices is digitized by the automatic pronunciation comparison algorithm 344, and is displayed on the learner computer screen via the pronunciation comparison result display 346 block.

In the server computer 300,
It has a native speaker audio signal database 302 and sends native speaker audio signals 342 to the learner computer 340 via the Internet in accordance with pronunciation practice scenarios requested by the learner.

Next, referring to FIG. 7B, the learner computer 340 sends the learner's voice signal 342 coming in through the microphone to the server computer 300 in the same manner as in the above-mentioned process, and learns from the native speaker voice signal database 302. A native speaker audio signal 304 to be selected is selected. Thereafter, the difference between the two sounds is digitized by the automatic pronunciation comparison algorithm 306, sent to the pronunciation comparison result display 346 of the learner computer, and displayed on the screen.

FIG. 8 is a flowchart showing a process of learning a foreign language pronunciation according to an embodiment of the present invention. Referring to FIG.
Step S400 in which the learner connects to the server computer via the Internet on his / her computer; Step S402 in which the learner inputs his / her information; Step S402 in which the learner desires from a plurality of pronunciation practice scenarios provided from the server. Selecting a scenario S404; listening to the sentence pronunciation of the native speaker from the selected scenario S406; and recording the learner's own pronunciation S
408; Step S410 of automatically calculating the difference between the two voices and displaying the comparison result on the screen; Step S412 of determining whether to continue the sentence pronunciation practice;
0, if it is determined that the user wants to interrupt the training, a step S414 of selecting whether or not to practice in another scenario is provided.

FIG. 9 is a flowchart showing a process of a verbal test for foreign language pronunciation according to an embodiment of the present invention, in which a learner connects to a server computer to test his / her foreign language pronunciation ability. It is a flowchart. Referring to FIG. 9, the learner connects to the server computer in step S5.
00; the learner inputs his / her information S502
A step S504 for the learner to select the difficulty level of a word or a sentence to be tested for pronunciation, and a step S for listening to the pronunciation of the native speaker for the selected difficulty level sentence.
506; the learner recording his pronunciation S508
Step S510 to check whether all sentences to be tested have been pronounced; and Step S512 to select whether or not to test again for other difficulty issues; Displaying the actual pronunciation comparison result on the screen.

[0046]

As described above, the foreign language pronunciation learning and oral test method using the automatic pronunciation comparison method on the Internet according to the present invention has the following advantages. First, by comparing the voice of the learner trying to learn the pronunciation of a foreign language with the voice of the native speaker by an automatic voice comparison algorithm using voice signal processing technology, the learner can objectively evaluate his pronunciation ability. There is a merit that a large evaluation numerical value can be understood.

Second, by providing a foreign language learning service in which the automatic speech comparison algorithm is provided in a web state, that is, a foreign language pronunciation learning and an oral test can be performed by a computer connected to the Internet, a language expert can be put on the side. Learners can verify their foreign language pronunciation ability by learning.

[Brief description of the drawings]

FIG. 1 is a block diagram of an automatic speech comparison network algorithm for learning a foreign language pronunciation and an oral test on the Internet according to the present invention.

FIG. 2 is a partial detailed view of the automatic audio comparison network shown in FIG. 1;

FIG. 3 is a detailed flowchart of comparing strength differences in the phoneme difference calculation model illustrated in FIG. 2;

FIG. 4 is a partial detailed view of the time-frequency dynamic warping board shown in FIG. 3;

5a to 5c are graphs showing a comparison of the phonemic difference calculation model shown in FIG. 2;

FIG. 6 is a schematic configuration diagram of an Internet system for foreign language pronunciation learning and an oral test according to the present invention.

FIG. 7 is a flowchart showing data processing between a learner computer and a server computer according to the present invention.

FIG. 8 is a flowchart illustrating a process of learning a foreign language pronunciation according to an embodiment of the present invention.

FIG. 9 is a flowchart illustrating a process of a verbal test for pronunciation of a foreign language according to an embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 10 learner speech signal 20 native speaker speech signal 30 automatic speech comparison network 40 DTW based difference comparison model 42 phoneme difference calculation model 44 prosody difference calculation model 50 error correction neural network 60 expert evaluation comparison network 70 error calculation network 300 server computer Internet 340 Learner Computer

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/00 G10L 3/00 533Z 15/24 539 // G10L 101: 027 551E 101: 18 571S F term ( Reference) 2C028 AA03 BA03 BB04 BB06 BC02 BD02 CA13 CB02 5D015 CC03 CC04 CC11 CC12 CC13 CC14 CC15 FF03 HH07 JJ00 KK02 LL13

Claims (8)

[Claims] 1. A foreign language pronunciation learning method for automatically comparing a native speaker's voice with a learner's voice and calculating the difference between the native speaker's voice and the native speaker's voice. A first step calculated by a base difference comparison network; a second step in which a difference between the calculated value of the DTW base difference comparison network and the comparison value evaluated by the expert evaluation comparison network is calculated by an error calculation network; And a third step of learning a DTW-based difference comparison network so that the first and second calculated difference values become smaller. A foreign language pronunciation learning and oral test method using an automatic pronunciation comparison method on the Internet, comprising: . 2. The DTW-based difference comparison network includes a strength difference comparison block for calculating linguistic pronunciation accuracy for each phoneme, syllable, word, and sentence, and a time difference comparison for calculating pronunciation duration difference. Block and formant
2. The method of claim 1, further comprising a phoneme difference calculation model comprising a frequency difference comparison block for calculating a difference in position. . 3. The DTW-based difference comparison network includes an intensity difference comparison block for calculating a relative position difference from a maximum peak value at a pitch contour line, and two at an end portion of pronunciation between a learner and a native speaker. And a rhythm difference comparison block calculated from the relative position and magnitude of peaks and valleys appearing between adjacent words and syllables. 2. A method for learning pronunciation of foreign languages and an oral test using an automatic pronunciation comparison method on the Internet according to claim 1, comprising a prosody difference calculation model. 4. The method for determining an automatic sounding comparison value comprises nonlinearly assembling six output values determined by the DTW-based difference comparison network via an error correction / correction neural network, 4. A pronunciation pronunciation learning method for foreign languages using an automatic pronunciation comparison method on the Internet according to claim 1, wherein an automatic pronunciation comparison value is calculated by learning so as to approach the comparative numerical value. And oral test methods. 5. The method of calculating the phoneme intensity difference comparison comprises extracting time-frequency domain features from audio signals of a native speaker and a learner, performing linear frequency conversion and DTW.
The time-frequency dynamic warping process of removing the characteristics of the speaker through the process and the difference between the frames of the feature vectors of the two voices are summed up by the Euclidean distance, and the linguistic strength difference of the phoneme is calculated. 3. A method for learning foreign language pronunciation and an oral test using an automatic pronunciation comparison method on the Internet according to claim 1 or 2, wherein the method is to obtain. 6. The learning process of the learner's foreign language pronunciation and the oral test, wherein the learner records his / her own voice on the learner's computer through a microphone, and the voice of the native speaker is the voice of the native speaker of the server computer. Brought from the signal database and listened to by the learner,
2. The method of claim 1, wherein a difference between two voices is calculated from a learner's computer by an automatic pronunciation comparison algorithm, and the result is displayed on a screen. Learning and oral test methods. 7. The learner's learning process of pronunciation in a foreign language includes the steps of: connecting the learner to a server computer via the Internet on his / her own computer; inputting his / her own information; Selecting the desired scenario from among several provided pronunciation practice scenarios; recording the learner's own pronunciation; automatically calculating the difference between the two voices and displaying the comparison result on the screen. 7. An automatic pronunciation comparison method on the Internet according to claim 6, comprising: determining whether to continue the sentence pronunciation practice; and selecting whether to practice in another scenario. A foreign language pronunciation learning and oral test method using 8. The home of the learner's foreign pronunciation learning oral test includes: a learner connecting to a server computer; a learner inputting his / her own information; and the learner attempting a pronunciation test. Selecting the difficulty level of the word or sentence; listening to the native speaker's pronunciation for the sentence of the selected difficulty level; recording the student's own pronunciation; and writing the text to be tested. Checking whether all pronunciations have been made; selecting whether to repeat the test for another difficulty level; and displaying the final pronunciation comparison result on the screen for the sound pronounced by the learner. 7. The foreign language pronunciation learning and oral test method using the automatic pronunciation comparison method on the Internet according to claim 6, characterized in that: