CN100514446C - Pronunciation evaluating method based on voice identification and voice analysis - Google Patents

Abstract

The pronunciation evaluating method based on voice identification and voice analysis includes the following steps: selecting input original phonetic signal, acquiring and converting into digital signal and frame dividing treatment; extracting characteristic parameter of the phonetic frame; identifying the input voice with voice identifying engine to obtain the sectional information of each word and/or syllable and calculating confidence of each word; and evaluating pronunciation truth of each word and/or syllable in input voice based on the confidence of each word and/or syllable. Further, the time length, energy and frequency information of each phonetic frequency may be calculated simultaneously and compared with those in standard pronunciation library, so as to calculate the similarity of each word and/or syllable, and the similarity may be weighted and added to the confidence to obtain the pronunciation truth. The present invention has greatly raised pronunciation evaluating precision.

Description

A kind of pronunciation evaluating method based on speech recognition and speech analysis

Technical field

The present invention relates to a kind of pronunciation evaluating method, more particularly, the present invention relates to pronunciation evaluating method based on speech recognition and speech analysis.

Background technology

" say " it is a important step in the language learning, existing a lot of language teaching softwares on the market, such as, English study software, the form of teaching that these software adopted substantially all is " recording contrast ", that is to say that the repeat playing that they can only provide the student to pronounce and teacher's example is pronounced allows student oneself listen contrast different orchids wherein and corrects one's pronunciation.In fact, the teaching efficiency that such mode can play is very limited, because the common cacoepy of people, just because of the difference of itself can not listening between " accurately " and " inaccurate " pronunciation.

At present, the English study software of pronunciation evaluation function has also appearred having in market, but they have mostly just adopted speech recognition technology to judge what the user said, can only provide a general judge, can only show the user this in short read OK, and cacoepy on certain or certain several words just often in the said a word of user, this each word (word) thought in can only can not accurately showing in short to the whether accurate method of assessing in short OK, allow the user correct targetedly, thereby be difficult to satisfy actual demand.

Summary of the invention

The technical problem to be solved in the present invention provides a kind of pronunciation evaluating method based on speech recognition and speech analysis, to the analysis of input voice and pass judgment on can word or individual character be unit, thereby improve the accuracy of assessment.In order to solve the problems of the technologies described above, the invention provides pronunciation evaluating method based on speech recognition and speech analysis, may further comprise the steps:

(a) choose the primary speech signal of input, gather and be converted to digital signal, carry out the branch frame then and handle;

(b) extract the characteristic parameter of speech frame, calculate the energy and the fundamental frequency of each speech frame, perhaps wherein a kind of;

(c) utilize speech recognition engine that these input voice are discerned, obtain each word (as, to English) or individual character (as, to Chinese) segment information;

(d) then with a kind of or combination in any in the speech frame energy, fundamental frequency and the segment information that obtain, compare with corresponding Received Pronunciation information, calculate a kind of or combination in any in the difference value of duration, energy and fundamental frequency of each word or individual character.

In the technique scheme, described step (d) is further divided into following steps:

(d1) obtain the duration information of syllable according to the speech frame quantity that each syllable comprised, the energy of all speech frames that each syllable comprises and fundamental frequency are obtained separately the fundamental frequency and the energy information of syllable after the addition divided by the number of speech frame respectively;

(d2) duration, energy and the fundamental frequency information of all syllables that respectively each word or individual character comprised are compared with corresponding Received Pronunciation information, calculate the difference value of duration, energy and the fundamental frequency of each word or individual character.

In the technique scheme, it is characterized in that, duration, fundamental frequency and the energy accumulation that also comprises all syllables according to each word or individual character in the described step (d1) obtains duration, fundamental frequency or the energy of each word or individual character, and compares with corresponding Received Pronunciation information and to obtain another group difference value of each word or individual character duration, energy and fundamental frequency.

In the technique scheme, when calculating the difference value of duration, energy and fundamental frequency of each word or individual character in the described step (d2), be respectively it to be comprised to get average after the squared difference of duration, energy and the fundamental frequency of all syllables and Received Pronunciation information and obtain.

The technical problem to be solved in the present invention provides a kind of pronunciation evaluating method based on speech recognition and speech analysis, can syllable be unit to analysis and the judge of importing voice, thereby improve the accuracy of assessment.In order to solve the problems of the technologies described above, the invention provides a kind of pronunciation evaluating method based on speech recognition and speech analysis, may further comprise the steps:

(o) choose the primary speech signal of input, gather and be converted to digital signal, carry out the branch frame then and handle;

(p) speech frame is carried out the extraction of characteristic parameter, calculate the energy and the fundamental frequency of each speech frame, perhaps wherein a kind of;

(q) utilize speech recognition engine that these input voice are discerned, obtain being accurate to the segment information of each syllable;

(r) with a kind of or combination in any in the speech frame energy, fundamental frequency and the segment information that obtain, compare with corresponding Received Pronunciation information, calculate a kind of or combination in any in the difference value of duration, energy and fundamental frequency of each syllable.

In the technique scheme, described step (r) is with duration, energy and the fundamental frequency of speech frame that each syllable comprises, the duration of this syllable that perhaps further calculates, energy and fundamental frequency are compared with corresponding Received Pronunciation information, obtain the difference value of each syllable duration, energy and fundamental frequency.

The present invention does not need the user to compare sound voluntarily, and is not only to judge with simple speech recognition technology what the user said, but utilizes speech recognition technology that accurate cutting is carried out in user's pronunciation earlier, is accurate to each word in a word, each syllable; Then, the pronunciation that comes the Accurate Analysis user at the strength information, frequency information, the prosodic information that utilize pronunciation has any difference with the pronunciation of standard, tell finally user (learner) problem goes out at which word, which syllable, and how to improve, can improve the degree of accuracy and the effect of pronunciation evaluation greatly.

Description of drawings

Fig. 1 is the system chart of the pronunciation evaluating method of the embodiment of the invention;

Fig. 2 is the exemplary plot as a result after segmentation discerned in certain example sentence;

Fig. 3 is the process flow diagram of first embodiment of the invention method;

Fig. 4 is the process flow diagram of second embodiment of the invention method.

Embodiment

Below in conjunction with accompanying drawing and embodiment the present invention is done and to describe in further detail.

First embodiment

Fig. 1 is the system chart of present embodiment pronunciation evaluating method, and as shown in the figure, the pronunciation evaluation system of present embodiment comprises pronunciation extracting module, speech recognition and align automatically regular module, information convergence analysis module.After the raw tone input, at first enter pronunciation extracting module and carry out feature extraction, this characteristic extraction procedure is that primary speech signal is carried out the data that features such as the intensity of phonation of each frame, pronunciation duration, fundamental curve were handled and obtained to the branch frame.Then primary speech signal is carried out the MFCC conversion, and enter sound identification module, to carrying out speech recognition through the voice signal after the MFCC conversion and alignment is regular automatically, match the segment information of an individual character that can be as accurate as each English word in the sentence or Chinese according to current learning content, received pronunciation model; Calculate the degree of confidence of each word or syllable more respectively according to these segment informations.Enter the information convergence analysis module at last, in this module, data such as the segment information that obtains in data such as the intensity of phonation that extracts in the comprehensive characteristics extraction module, pronunciation duration, fundamental curve and the sound identification module, degree of confidence, and convergence analysis is carried out in contrast learning content sample storehouse, draw the final judge to this pronunciation, this judge is accurate to each word or the individual character of this pronunciation.

Fig. 3 is the process flow diagram of present embodiment, as shown in the figure, may further comprise the steps:

Step 100, the user reads a word in the learning content;

Step 101 is carried out the primary speech signal collection, and the analog-signal transitions of each user pronunciation is become digital sampled signal;

Step 102 is carried out the branch frame with the digital signal of voice and is handled, and is the length of an analysis frame with 25ms usually, and after each frame analysis was intact, with the mobile backward 10ms of analysis frame, re-treatment again finished until all signal Processing;

Step 103 calculates the speech energy of this frame, MFCC parameter, and fundamental frequency respectively to each frame voice signal;

Step 104 according to current learning content, utilizes speech recognition engine that the input voice are discerned, and obtains at this segment information in short, and present embodiment is to obtain each word or individual character, and the segment information of each word or syllable that word comprises; Fig. 2 discerns exemplary plot as a result after the segmentation to certain example sentence.

Step 105, obtain segment information after, continue to calculate the degree of confidence of each word in the input voice or individual character;

In the above-mentioned steps 104 and 105, patented claim is all adopted in the calculating of segment information and degree of confidence: " speech recognition confidence evaluation method and system and use this method dictation device (application number: 02148686.7) " in disclosed algorithm, but can adopt other algorithm, the present invention is not limited this yet.

Step 106, according to speech energy, the fundamental frequency of the segment information that obtains and each frame, duration, energy and the fundamental frequency of each speech or word in the calculating input voice;

Word or individual character may be made up of one or more continuous syllables, present embodiment is duration, energy and the fundamental frequency that calculates each syllable earlier, obtains duration, energy and the fundamental frequency of each speech or word again according to the above-mentioned information of the syllable that is comprised in each speech or the word.

The duration of syllable is exactly the pronunciation length of each syllable, can directly obtain according to segment information, and frame voice are equivalent to 1oms, and the number of all speech frames that each syllable comprises just can be represented the duration of this syllable; The speech energy of syllable and fundamental frequency are then by obtaining averaging after the speech energy of all speech frames of this syllable and the fundamental frequency accumulative total.Energy on each frame of user's raw tone is designated as eng (i), fundamental frequency are designated as pitch (i), duration is designated as dur (i), wherein i represents frame number; Energy on certain syllable is designated as eng (k), fundamental frequency are designated as pitch (k), duration is designated as dur (k), wherein k represents the sequence number of syllable.Then dur (k) equals the number of the frame that comprises among the syllable k; Energy eng (k) is each frame energy eng (i) addition that syllable k is comprised, and the number dur (k) divided by the frame that this syllable comprised obtains again; Fundamental frequency pitch (k) is that the frame number dur (k) divided by syllable k obtains again with fundamental frequency pitch (i) addition of each frame in the syllable k.

In the present embodiment, the duration of each word or individual character, energy and fundamental frequency obtain by duration, energy and the fundamental frequency stack of the syllable that it comprised.The energy of each word or word in the sentence is designated as eng_w (j), fundamental frequency are designated as pitch_w (j), duration is designated as dur_w (j); Wherein j represents the sequence number of word or individual character, then duration dur_w (j) equal all syllables that word or word j comprise duration dur's (k) and; Energy eng_w (j) equal all syllables that word or word j comprise energy eng's (k) and; Fundamental frequency pitch_w (j) equal all syllables that word or word j comprise fundamental frequency pitch's (k) and.

Step 107 is compared Received Pronunciation information corresponding in degree of confidence, duration, energy and the fundamental frequency information of each speech that obtains or word and the learning content sample storehouse, calculates the pronunciation validity of each speech or word;

The specific algorithm of present embodiment is as follows:

1) finds earlier in the learning content sample storehouse and the corresponding pronunciation example sentence of this input voice content, obtain one group of duration, fundamental frequency, energy corresponding " standard " duration, fundamental frequency and energy information with each syllable of calculating and each speech (word), be designated as respectively: dur0 (k), eng0 (k), pitch0 (k) and dur_w0 (j), eng_w0 (j), pitch_w0 (j).This standard duration, fundamental frequency and energy information can be preserved in advance, also can calculate in real time according to this example sentence in received pronunciation storehouse.

2) calculate duration, the energy of each word or word j, the difference value of fundamental frequency then:

$\begin{array}{cc}\mathrm{\Δdur}\left(j\right)=\underset{k}{\mathrm{\Σ}}{(\mathrm{dur}\left(k\right)-\mathrm{dur}0\left(k\right))}^2/N& \mathrm{for\; all\; syllable}\left(k\right)\∈\mathrm{word}\left(j\right)\end{array}$

$\begin{array}{cc}\mathrm{\Δeng}\left(j\right)=\underset{k}{\mathrm{\Σ}}{(\mathrm{eng}\left(k\right)-\mathrm{eng}0\left(k\right))}^2/N& \mathrm{for\; all\; syllable}\left(k\right)\∈\mathrm{word}\left(j\right)\end{array}$

$\begin{array}{cc}\mathrm{\Δpitch}\left(j\right)=\underset{k}{\mathrm{\Σ}}{(\mathrm{pitch}\left(k\right)-\mathrm{pitch}0\left(k\right))}^2/N& \mathrm{for\; all\; syllable}\left(k\right)\∈\mathrm{word}\left(j\right)\end{array}$

Wherein: N is the number of the syllable (syllable) that belongs to Word (j), Δ dur (j), and Δ eng (j), Δ pitch (j) is respectively the duration difference value of single speech or word j, capacity volume variance value, fundamental frequency difference value.

3) again the information of duration, energy, fundamental frequency difference value is merged, calculates the similarity of these information:

Parameter $a=\sqrt{w_1\·\mathrm{\Δdur}\left(j\right)+w_2\·\mathrm{\Δeng}\left(j\right)+w_3\·\mathrm{\Δpitch}\left(j\right)}---\left(1\right)$

Parameter $b=0.5*\sqrt{w_1\·x+w_2\·y+w_3\·z}---\left(2\right)$

Wherein: x=|dur_w0 (j)-dur_w (j) | ²Y=|eng_w0 (j)-eng_w (j) | ²Z=|pitch_w0 (j)-pitch_w (j) | ²

Scoring functions is: sigmo (x)=arctan (x-3.5)/π * 100+50 (3)

The substitution formula as a result (3) of formula (1), (2) formula is obtained similarity:

score ₁＝sigmo(-log(a+b)/2)

Wherein: w1, w2, w3 represent the weight to different information, can be provided with respectively, as be 0.5,0.3,0.2.

4), calculate the pronunciation validity scoring of each word or word at last according to the degree of confidence of each word or word and the similarity of calculating above.

The degree of confidence of the word (j) that obtains is designated as score ₂, so, the pronunciation validity of final word (j) is exactly score ₁And score ₂Weighted mean:

score＝w ₁·score ₁+w ₂·score ₂

Both weights can be made as identical, that is: w ₁=w ₂=0.5.Can certainly be made as difference.

Step 108 is calculated the validity of whole word, equals the mean value of the validity score of all words in the word or word.

Step 109 is made judge according to above result of calculation to this pronunciation, and the validity of validity, each word or the individual character of whole word is fed back to the user.

Second embodiment

The key distinction of the present embodiment and first embodiment is that present embodiment has also carried out the pronunciation validity assessment of syllable, and to point out the difference of user pronunciation and Received Pronunciation more accurately, in addition, present embodiment has adopted different scoring methods.

Present embodiment the pronunciation evaluation system identical with first embodiment, its method flow may further comprise the steps:

Step 200, the user reads a word in the learning content;

Step 201 is carried out the primary speech signal collection, and the analog-signal transitions of each user pronunciation is become digital sampled signal;

Step 202 is carried out the branch frame with the digital signal of voice and is handled, and is the length of an analysis frame with 25ms usually, and after each frame analysis was intact, with the mobile backward 10ms of analysis frame, re-treatment again finished until all signal Processing;

Step 203 calculates the speech energy of this frame, MFCC parameter (also can be other characteristic parameter), and fundamental frequency respectively to each frame voice signal;

Step 204 according to current learning content, utilizes speech recognition engine that the input voice are discerned, obtains at this each word or individual character in short, and the segment information of each syllable;

Step 205, obtain segment information after, calculate each word or word, with and the degree of confidence of each syllable of being comprised; Still adopting application number is 02148686.7 the disclosed algorithm of patented claim;

Step 206, duration, energy and fundamental frequency information dur (i) with each frame of obtaining, eng (i), the corresponding Received Pronunciation information of this frame: dur0 (i) in pitch (i) and the learning content sample storehouse, eng0 (i), pitch0 (i) compares, and calculates the similarity of duration, energy and the fundamental frequency of each syllable in the input voice:

Calculate duration difference value, capacity volume variance value and the fundamental frequency difference value of syllable k: Δ dur (k) earlier, Δ eng (k), Δ pitch (k):

$\begin{array}{cc}\mathrm{\Δdur}\left(k\right)=\underset{i}{\mathrm{\Σ}}{(\mathrm{dur}\left(i\right)-\mathrm{dur}0\left(i\right))}^2/N& \mathrm{for\; all\; frame}\left(i\right)\∈\mathrm{syllable}\left(k\right)\end{array}$

$\begin{array}{cc}\mathrm{\Δeng}\left(k\right)=\underset{i}{\mathrm{\Σ}}{(\mathrm{eng}\left(i\right)-\mathrm{eng}0\left(i\right))}^2/N& \mathrm{for\; all\; frame}\left(i\right)\∈\mathrm{syllable}\left(k\right)\end{array}$

$\begin{array}{cc}\mathrm{\Δpitch}\left(k\right)=\underset{i}{\mathrm{\Σ}}{(\mathrm{pitch}\left(i\right)-\mathrm{pitch}0\left(i\right))}^2/N& \mathrm{for\; al\; lframe}\left(i\right)\∈\mathrm{syllable}\left(k\right)\end{array}$

Wherein: N is the number of the frame (frame) that belongs to syllable (k).Calculate the similarity of duration, energy and fundamental frequency then respectively:

${\mathrm{score}}_a=\mathrm{sigmo}(-\log (\sqrt{\mathrm{\Δdur}\left(k\right)}+|\mathrm{dur}0\left(k\right)-\mathrm{dur}\left(k\right)|/2))$

${\mathrm{score}}_b=\mathrm{sigmo}(-\log (\sqrt{\mathrm{\Δeng}\left(k\right)}+|\mathrm{eng}0\left(k\right)-\mathrm{eng}\left(k\right)|/2))$

${\mathrm{score}}_c=\mathrm{sigmo}(-\log (\sqrt{\mathrm{\Δpitch}\left(k\right)}+|\mathrm{pitch}0\left(k\right)-\mathrm{pitch}\left(k\right)|/2))$

Sigmo wherein (x) function is identical with first embodiment.

Step 207 is calculated the pronunciation validity of each syllable that is comprised in each speech or the word; Be that similarity according to the degree of confidence of each syllable and duration, energy, fundamental frequency is weighted on average:

Score=w ₁Score ₁+ w ₂Score ₂+ w ₃Score ₃+ w ₄Score ₄Wherein: w1, w2, w3, w4 represent the weight to different information, can be made as 0.25,0.15,0.10,0.5 respectively;

Step 208 calculates the validity of single speech or word, and the validity of word or individual character is determined by the average of its all syllable validities that comprise, and also can adopt the method for first embodiment to calculate;

Step 209 is calculated the validity of whole word, equals the mean value of the validity score of all words in the words or individual character;

Step 210 is made judge according to above result of calculation to this pronunciation, and the validity of whole word and each syllable is fed back to the user.

The present invention can have a lot of conversion on the basis of the foregoing description, for example the validity for each speech or word is to have considered degree of confidence, duration, fundamental frequency and four information of energy, wherein duration, fundamental frequency and energy can not considered yet, perhaps consider wherein a kind of or its combination in any.

And for example, for word or individual character, and two kinds have been enumerated above the concrete computing method of the validity of syllable, but much conversion of this algorithm, for example when first embodiment calculates duration, fundamental frequency and the energy of word or individual character and difference value thereof, do not calculate the difference value of its duration that comprises syllable, fundamental frequency and energy, and the frame that directly adopts this word or individual character and comprised calculates; The computing method of similarity and last validity can have a variety of, or the like.

Claims (4)

1, a kind of pronunciation evaluating method based on speech recognition and speech analysis may further comprise the steps:

(a) choose the primary speech signal of input, gather and be converted to digital signal, carry out the branch frame then and handle;

(b) extract the characteristic parameter of speech frame, calculate the energy and the fundamental frequency of each speech frame, perhaps wherein a kind of;

(c) utilize speech recognition engine that these input voice are discerned, obtain the segment information of each word or individual character;

(d) then with a kind of or combination in any in the speech frame energy, fundamental frequency and the segment information that obtain, compare with corresponding Received Pronunciation information, calculate a kind of or combination in any in the difference value of duration, energy and fundamental frequency of each word or individual character; Described step (d) is further divided into following steps:

(d1) obtain the duration information of syllable according to the speech frame quantity that each syllable comprised, the energy of all speech frames that each syllable comprises and fundamental frequency are obtained separately the fundamental frequency and the energy information of syllable after the addition divided by the number of speech frame respectively;

(d2) duration, energy and the fundamental frequency information of all syllables that respectively each word or individual character comprised are compared with corresponding Received Pronunciation information, calculate the difference value of duration, energy and the fundamental frequency of each word or individual character.

2, pronunciation evaluating method as claimed in claim 1, it is characterized in that, duration, fundamental frequency and the energy accumulation that also comprises all syllables according to each word or individual character in the described step (d1) obtains duration, fundamental frequency or the energy of each word or individual character, and compares with corresponding Received Pronunciation information and to obtain another group difference value of each word or individual character duration, energy and fundamental frequency.

3, pronunciation evaluating method as claimed in claim 1, it is characterized in that, when calculating the difference value of duration, energy and fundamental frequency of each word or individual character in the described step (d2), be respectively it to be comprised to get average after the squared difference of duration, energy and the fundamental frequency of all syllables and Received Pronunciation information and obtain.

4, a kind of pronunciation evaluating method based on speech recognition and speech analysis may further comprise the steps:

(o) choose the primary speech signal of input, gather and be converted to digital signal, carry out the branch frame then and handle;

(p) speech frame is carried out the extraction of characteristic parameter, calculate the energy and the fundamental frequency of each speech frame, perhaps wherein a kind of;

(q) utilize speech recognition engine that these input voice are discerned, obtain being accurate to the segment information of each syllable;

(r) with duration, energy and the fundamental frequency of speech frame that each syllable comprises, the duration of this syllable that perhaps further calculates, energy and fundamental frequency are compared with corresponding Received Pronunciation information, obtain the difference value of each syllable duration, energy and fundamental frequency.

Images