TWI294107B - A pronunciation-scored method for the application of voice and image in the e-learning - Google Patents

Description

1294107 IX. Description of the invention: [Technical field to which the invention pertains] The present invention relates to a method for sound evaluation using sound and video in digital learning, and more particularly to a method of visualizing a learner in a pronunciation process. In the middle, the change of the lip shape is detected to be used for comparison with the teacher to display the position 2 where the pronunciation may be wrong and the learner's voice is recorded for subsequent use in making the sound. And the score between the two. Known [Prior Art] In recent years, more and more research has been done on lip recognition. The most research is on the part of the lip contour, such as the way through the lip pattern = contouring, first different types of lips - Classification, with different Lip Contour Patterns, mathematically or whatever the lips are used to obtain general lip contour information, and then the fuzzy clustering method can be used to compare the lips. Which lip contour template is mapped, and the corresponding lip wheel waste template can be changed into the lip to extract the lip contour. In addition, digital learning is becoming more and more popular because of the increasing use of digital learning and the increasing popularity of computer use. For example, there are many computer-aided teaching software (CAD), which allows learners to learn through the operation process through a graphical introduction. There are many English teaching software that provide listening, speaking, reading, and writing teaching and practice functions, but there are still many things. The English teaching software often only provides teaching videos or real human voices in the "speaking" part. To judge the correctness of the learner's pronunciation is a big challenge for the learner to practice the pronunciation. If you simply use the "real heart", you will get half the work for the learner. However, for the hearing impaired, 5 1294107 mouth English " my pronunciation teaching is not feasible, because they can not learn the language from the teacher's pronunciation. In addition, many non-synchronous online teachings often only provide simple teaching videos. The video is sent to the client through the multimedia cage, and there are many problems in the use of the voice teaching. For example, the learner cannot know whether his or her pronunciation is correct, because it is simply = listening to the teacher's pronunciation or sentence f in the teaching film. If you follow the practice f, you can't know whether the pronunciation is correct. Second, it is impossible to know the part where the pronunciation is wrong. If the pronunciation of the instructional film is continuously heard, the pronunciation can only be changed similarly, but it is impossible to know that there is an error in the lip shape of the sound syllable. Second, the hearing impaired cannot learn pronunciation from these English teaching sentences. In addition, in the aspects of listening, speaking, reading and writing in speech training, "saying, evaluation and learning tools" is a very important but difficult part of the work. Some studies at home and abroad are directed at the voices spoken by practitioners. Characteristics are compared and scored, but the use of multimedia features, such as lip shape changes and comparisons, is still rare. Alignment through the contours of the lips makes it easy for some lips to find similarities from the template or It is the conformity of the contour, which leads to the possibility of misjudgment or error. In recent years, the use of lip visual information to confirm the speaker's words has attracted the interest of many researchers. If there is external noise interference in the environment where the sound recognition is located, It tends to cause a significant drop in the recognition rate, but the right side is assisted by the information of the lip and lip type, which can effectively increase the accuracy. In order to change the contours and colors of the lips, the lips will be picked up because of the complexion of the face and the color of the lips. The weaker the contrast intensity, the more difficult it is. The current method of lip image segmentation can be divided into two types, the first type of lip The cutting method is directly obtained through the color space. The method of this method is to convert the RGB color into another color system to enhance the contrast between the lips and the skin color. The processing time of the method is relatively fast, but since there is a lot of noise after the conversion, it is difficult to distinguish the contour boundary of the lip when the color contrast is not obvious. Therefore, the Markov random field technique is used. It can be used to reduce the segmentation error caused by noise after color conversion. The second lip segmentation method is referred to as FCM by fuzzy C-means clustering, which is based on " C-means algorithm Derived grouping method, Bezdek first proposed this method in 1973. It is a data grouping technique that uses the value of fuzzy attribution function to indicate the extent to which each point of data belongs to a cluster center. The distance from a known point to any cluster center multiplied by the value of the attribution function at that point as the objective function, and The minimization of the objective function reaches the purpose of judging the cluster. However, the fuzzy C-means grouping method is also affected by the intensity of color contrast, so a new fuzzy algorithm fuzzy c-means with shape function (FCMS) has been proposed. The operation of enhancing the spatial distance has the color information contrast supplement # strong, because the similar color of the lips may not be concentrated, or the lip block is not complete after being taken, so the Elliptic Shape Function can be used. Positioning the lip range, and calculating the block circle, has obtained a more accurate lip contour, and can automatically use the lip function to apply to the continuous image, which can be used to find the lips one by one. The outline range, which is a method of dividing a lip into a continuous RGB format image, then extracting it by lip characteristic parameters, and recognizing it using a hidden Hovden Markov Model. 7

1294107 Through the fuzzy c-means with shape function (FCMS) to capture the lip contour, the lip range can now be detected correctly, but because of the more complicated mathematical operations to circle the contour, the time it takes will often come. More. In terms of speech scoring, domestic related researches include: (1) Using the three characteristic parameters of sound [Magnitude, Pitch c〇nt〇ur, and Mel Cepstrum parameters (Mel) ~Frequency Cepstral Coefficients)] and the dynamic time waning (DTW) and Hidden Markov Model (HMM) methods are used to score. The results of the test show that the Melt's cepstrum parameter represents the same weight. The most important is the high, followed by the base frequency, and finally the volume intensity curve. (-) Cut through the English voice signal to provide a method for cutting the voice signal into the parent cause, the time segment' and using the pre-trained two English/text pronunciation acoustic models as the comparison criteria. The appropriate acoustic model is then tailored to the correct pronunciation segment by voice recognition techniques in different native languages. The scoring method used is to compare the similarity between standard speech and scoring speech, and adopt four characteristic parameters: strong volume f j fundamental frequency detour curve, vocalization and gradual change, and _ log probability difference. Rong Bao: The relevant domestic research on music training includes: teaching internal sound and rhythm rhythm. The learning method is the main practice activity through the oral and identification rules. In addition, 'foreign in the fusion sound Pandan ✓ You / audio-visual system in the speech recognition side ° t blmodal, the research on the recognition of lip recognition to gain attention to the importance of the lip image 1294107 is divided into contour-based and image-based. Contour-based research uses edge information, deformable templates, or active contours to obtain feature values. The biggest advantage of this method is that the eigenvalues remain unchanged in spatial translation, rotation, scaling, and illumination. However, the contour-based approach loses a lot of useful information, such as the appearance of teeth and tongue. At present, the automatic speech recognition system (ASR) can distinguish the s_h of the speaker at the front end, but since the automatic speech recognition is purely susceptible to buzzing interference outside the environment, the S is such that the speech _ rate is changed. low. [Summary of the Invention] The invention mainly proposes an innovative image. The continuous lip type image is scored in English teaching through the comparison of %, type and sound to enhance the language learner-L visual learning interface. . In a visual way, the learner's change in ==== is used to score with the teacher. The following are for the learner, the teacher, the measurement: the system, the death, and so on.) Learner: The better for the learner, the simpler by two or two == correct. Assisting the hearing-impaired people in learning ▲ their own 曰疋 予 _ bg process, through the figure (9 1294107 fruiting way, to effectively point out the need to improve the pronunciation of the part can make the real teaching - Layer auxiliary effect -), scholar: Provides the original image and sound file of the lip picture recording the speaker's pronunciation, so that the teacher can freely expand the English pronunciation practice sentence, just by recording the new lip picture original file, you can let Learners practice new pronunciation sentences to benchmark. (2) System: In terms of system, it will capture the continuous picture of the lips of the pronunciation learner and the right to sound, and perform the action of scoring the series by the pre-recorded voice of the teacher and the accompanying system. To present the results of the 5 flat assessments, as well as the differences between the display and the learners' pronunciations, 'allowing learners to know more about the departments that need to be strengthened—because E-Learni ng is currently used in many tutors, schools, etc. , = can be used effectively 'can reduce the cost of many personnel.尤绕 = learner's 'if you can provide sound and visual way to =3⁄4 Yangyu pronunciation' to visualize the continuous lip picture to show the position of the wrong pronunciation; through the sound to leak the learner to practice the English sentence pronunciation, And it is more effective to pronounce the pronunciation of the learner's pronunciation = and the advantage of the visual way is not only to show the wrong position, but also to assist the hearing impaired in the process of learning English. There is a more convenient aid to help. The characteristics of the interactive English teaching proposed by the present invention can be enumerated as follows: 1. The function of providing red pronunciation is sufficient: the main material solution f is practicing the pronunciation... determining whether the lip shape can conform to the lip shape of the teacher, and let the learner Can 10 1294107 Know enough, which syllables need to strengthen the practice. 2. Assisting the hearing impaired to learn the language: Because the hearing impaired learn the biggest obstacle of f language, it is impossible to clearly hear the sound from the pronunciation to learn. _ Lip recognition to help the hearing impaired in language learning, by observing the continuous picture of the lips, to gradually learn and ^ = ^ to let the hearing impaired know (4) whether it is with the teaching face ^ ^ 士 people with obstacles, It is possible to understand the correct rate by the lip of the lips; the force and the scoring mechanism. Fantasy Island 3. == The present invention proposes an innovative scoring mechanism in which it is possible to know which range of scores is large (4). 4. = Sound and skill to assess the pronunciation accurately ^ The current neighboring technology of Qianyin is full of maturity, but if there is interference in the surrounding environment, the recognition ability will be greatly reduced, destroying ^ = accuracy. The correction of the continuous lip image enhances the accuracy of the sound, not only in a visual way, but also complements the positive rotation of the sound by external dryness, combined with sound and image. Learning the pronunciation teaching software, most of them use the recording mode to let one hunter to learn f repeatedly, but it is difficult for the learner to know whether the pronunciation is correct or not. Therefore, the present invention is said by humans. (4) The size and outline of the mouth are compared, and the pronunciation is found to be mixed, and can be taken as ^^. The present invention can be roughly divided into the following functions: the first 2 I tone? Scholars can first record their own pronunciation (4) and sound through ^; Gong 1 will record each sentence of English, and make a key picture of the lips to capture ey f two me) to find out the important key lips, and later compare with the learner Yes, first, for the learner, just use WebCam to align the lips of 1294107 to record, and the secret will take the recorded picture into a picture by itself, and then circle the part of the lips, and the mouth of the teacher. Chen, do comparison, to know the correctness and score of the pronunciation; third, through the Connected Component Analysis (Automatically detect the mouth profile, and effectively find the range of lip characteristics; fourth, English hair Li Er 2 is updated by the instructor, and the continuous lip map # 2 = down, and the even (four) parameters are obtained; fifth, the present invention adopts the method in the evaluation of the image by combining the sounds. To score, you can improve accuracy. 4 (four) source 'The system used in the present invention can be divided into several, English teacher, sentence database 'scoring machine; one layer of the middle face · learning learner' by Webcam to extract the learner's pronunciation and mouth sound 'And read the teacher's lips and voice from the statement database; the teacher loses the supply - the function of the English teaching recording, the English old mouth cr, even, and the shirt is recorded to the 丨 consultation and can be repeated recording - Sentence English; U丨贝枓库内 '辰型' mainly stores the currently recorded English teaching sentences and provides the results of the future search for Wei, Ρ Mainly assesses the learner's pronunciation accuracy and ΐ 槛The sub-mechanism comes with a score of 8 and 2; the invention is evaluated by means of a brooding mechanism relative to sound and image. [Embodiment] For a more complete and clear disclosure of the technical content, the object of the invention, and the effect of the 1294107 which are utilized by the present invention, reference is made to the drawings and drawings, which are illustrated in the following, and Please send one: send sound and image in the pronunciation evaluation method of digital learning, and show that the learner detects the change of the lip shape during the pronunciation process' to compare with the teacher. Yes, there is no possibility of a wrong position in the pronunciation process, and the learner's voice = recorded 'for subsequent use to score between the sound and the image. The process steps of the present invention include the following: (S1) the singer adds only the English sentences to be added by this interface, and the location to be stored, and then writes the paths and sentences to the syllabus. In the database, for later reading, as shown in the second figure. (b) Select the sentence (S2) to provide the sentence that the learner wants to select the exercise [see the third picture], or by providing it on the window. The search function is used to find sentences to be practiced, and in the "All sentences" option, the system can automatically list all previously stored sentences and sort them according to the order of the letters. (c) WebCam Capture (S3) Learning The teacher/teacher will point his/her lips to the WebCam and press the capture button. The system will automatically capture the image in the picture into a single lip image [see the fourth picture]. (d) Automated lip interval selection (S4) The present invention uses lip automatic detection to obtain lip information, and uses lip information to apply to spoken language teaching. The color space YCbCr is used to cut and first find the face skin, and then The skin is automatically detected by the RGB color space. 13 1294107 The fifth picture shows the process of automatic lip detection: First, face range detection, in order to accurately correct the lip block and speed up the processing, first Through the YCbCr face state % < between the face part, and then the RGB, HSV color cut in the range of the face to determine the possible block of the lips, and in order to _,, | Chu get the lip area Block, so after the lip color cutting, the subsequent morphological operation and the intermediate value filter for the noise processing, then the binarized image after the correction is given the same number 2 by the numbering operation, and then find the maximum The block is the lip block. (1) Face Range " 贞 贞 (S41) In order to effectively detect the range of the lip block, the present invention first pre-processes the face and cuts it through YCbCr. Skin color, after YCbCr conversion, the value of Cb is between [77,130], the value of Cr is between [130,173], and the resulting image is shown in the sixth figure, and then the skin area after cutting is found. Upper and lower face , the left and right four borders (11), (12), (13), (14), because the lip image captured through the WebCam is the range below the nose, so the leftmost boundary needs to be 13) and the rightmost boundary (14) to do the retracting action, that is, the left and right indented one eighth is regarded as the critical line of the leftmost boundary (13) and the rightmost boundary (14), if the search range exceeds this The critical edge is bounded by one-eighth of the indentation. When the face range boundaries 01), (12), (13), (14) are obtained, the required lip range information can be searched within this range. (2) Lip color cutting (S42) 14 1294107 In the present invention, in order to enable automatic lip detection, it is convenient to obtain lip characteristic information more accurately when scoring, and in general, the main color of the face and lips Most of them are red, so in the present invention, other lipsticks are not discussed, and the color of the lips is different. Through the color information, the lip range block can be obtained. In the color cutting lip block, the present invention cuts out the red block range and the Cb and Cr of the YCbCr color space by the color information of the R value and the G value of the RGB color system.

1. Cut through the RGB color system [Refer to the seventh picture]: Since the color of the lips is biased to red, the image can be cut by the R value and the G value of the RGB color system, and the relationship between red and green. The relationship is as follows: 1) (2)

Aim ^lim 0 <r- otherwise

Where R is the red value of the RGB color system, G is the green value, and Liim is the minimum threshold, which is the maximum upper limit of the threshold. If the ratio between the R value and the G value is within the interval, set to 1 Otherwise, it is 0, and the lip block is detected by converting into a binarized image to use the block range to determine the center coordinate position of the lips and the width and height of the lips. In the current test, the lip block can be distinguished between 1.1 and 2. 2. Cutting through HSV color space: In the HSV color system, Η 15 1294107 is expressed as hue value (Hue), S is expressed as saturation (Saturation) and V is expressed as brightness (Value), and HSV can be quantized into eight kinds. The colors are red, yellow, green, cyan, blue, magenta, white, and black, with the range of tonal values being [〇, 359], and the range of saturation and brightness being [0·0 , 1· 〇], the color can be clearly cut through the HSV color system. Therefore, in order to cut the lip color block, the original lip image is first converted into HSV, and the range of Η is set to be greater than 300 to less than 60, and the range of S is between [0·16, 0·85], and V The value is between [0·21, 0·85], and for each pixel (ρ), it is judged whether the value of each of them is in the above range, and if so, the prime point is set to 1, otherwise Cheng Yu. The eighth picture shows the results found through the HSV. The lip blocks cut by the two different color systems have their own advantages and disadvantages, and the RGB color system is susceptible to the brightness of the light. Therefore, the present invention combines the two color systems to perform lip block extraction to improve accuracy. Sex. (3) Noise removal (S43) Since the image of the lips is subject to a lot of light and the hardware itself, there is a lot of noise in the binary image (Binary Image) produced by lip color cutting. 'And may be because other colors may fall into the same range of values, the cut lips block is more than one piece, so the present invention does not need to be removed by Morphology Operation and Median Filter 16 1294107 The desired block range. 1. Morphological operation·· is a type of image that can be binarized and analyzed to obtain images of different shapes. The commonly used basic arithmetic elements include: Er〇sion , Dilation, Disconnect (0pen), and Closure (Cl0se). Through the morphological operation, the image of the lip cut by the rgb and color system is broken and closed, and some extra noise can be removed and the small gap can be filled, so that the lip contour is more obvious. 2 = = value filter n: After the morphological operation of the break and close the transport, in order to remove the remaining noise, through the 3X3 size median filter (Median Filter) to remove the image noise, if here More than half of the pixels in the 3X3 matrix are white (P) is white, then the nine pixels (p) 7 of this mask are changed to white. This method can remove the partial and scattered noise to reduce the noise. The error rate of lip block capture. The ninth figure shows the result of noise removal after the disconnection and closing of the type operation, and then through the intermediate value filter. (4) Lip block capture (S44) ), the mouth through the above lip color cutting and noise removal, can = area ^ may be the contour of the lips, because the lip image of the continuous image when the speech is captured, will be limited through the WebCam 掏, the vicinity of the lip, this It is because (4) clearly pass the lip change information for the score estimation. If the WebCam captures the upper body to draw, it is relatively difficult to distinguish the size of the lip change of the mouth 17 1294107, so the present invention only captures The lips Closed range is used for scoring. Connected Component Analysis is a method for distinguishing the boundaries of objects in an object image and the composition of regions. You can assign a number to the different blocks in the image to distinguish the The image is composed of which blocks. For the numbering algorithm, the present invention takes numbers in units of pixels (P) (pixel), as follows: Each pixel (P)P in the picture The neighboring edge can be divided into four neighboring edges or eight neighboring edges. If the four neighboring edges are pixel points (P) 1, 2, 3, and 4 points, if there are eight neighboring edges, there are 1, 2, and 3, 4, 5, 6, 7, 8 adjacent pixels (P), as shown in the tenth (a) and (b).

In this case, the four neighboring edges are the same block, and the neighboring edges of the pixel (Ρ) are the same block. The binary image after the intermediate value filter is judged from the upper left point to the lower right. Point, each point is judged one by one by recursively, and the search order of the pixel (Ρ) is up, left, down, right, and when the end of the recursion means that a block has been marked and the next pixel is Point (Ρ) to search, if it is already = the number is skipped to the next search and so on, etc., after the prime point (Ρ) has been judged, then all the blocks are also completed. The eleventh picture is the original image, and the twelfth picture is the edited image. After numbering each block by numbering algorithm, it can be known that the block is cut by the lip color j by cutting through the RGB color system, so the present invention uses the block r number. Searching and representing the lip block, looking for the brother f is the search from the upper left point of the image, if the first point is found (that is, the value of the pixel (p) is the number with the largest number of blocks), the representative is The top position of this block records the coordinates of this point, and then continues to search for the lowest lip contour: Material 'When the upper and lower boundaries are found, the left and right edges of the lip contour are also found' algorithm:

Fqi" i = 〇 to picture height F〇r j = 0 to picture width

If pixel(i,j) = the number with the largest number of blocks. Then top = i < top ? i : top bottom = i > bottom ? i : top left = j < left ? j : left right = j > Right ? j : right End If • Next j

Top, bottom, left, and right in Next i represent the uppermost boundary (11), the lowermost boundary (12), the leftmost boundary (13), and the rightmost boundary (14) of the lip contour, respectively. The thirteenth image is the contour. method. When the uppermost boundary (11), the lowermost boundary (12), the leftmost boundary (13), and the rightmost boundary (14) of the lip rim are obtained, the center position (0) and width (W) of the lip can be obtained. High 19 1294107 [as shown in Figure 14], and these can be used as simple information for the lips. (e) Scoring mechanism (S5) The scoring mechanism of the present invention is a mechanism for visualizing lip scores and lip-to-mouth combined sounds. In the lip scoring mechanism, the shape of the lip changes in human pronunciation is compared with the standard pronunciation lip to evaluate the difference; in the combined lip and sound scoring mechanism, the current sound recognition rate is mature. However, since the accuracy of the data is greatly reduced when the environment has external sound interference, the lips and the sounds are mutually assisted so that they can be correctly scored under external sound interference. 1. Visualized lip score: In the visualized lip score, the present invention uses a Dynamic Time Warping (DTW) algorithm in conjunction with Pattern Matching to pass the standard pronunciation of the teacher's lips. The similarity or difference between the picture and the learner's lip picture, filtering out the need to delete the extra picture or retain the important key face to calculate the score, if the higher the degree, the score obtained The higher. Dynamic Time Warping (DTW) is often used to solve the problem of speech recognition. Because the time of each sentence is different, in order to compare the similarity between the two, the sound of the test data and the sound of the reference data Obtain a nonlinear response with the smallest error [as in the fifteenth figure]. However, in the sixteenth figure, through dynamic time warping (DTW), it can be found that the length of the pronunciation of A and B may be inconsistent, so DTW can Correspondence of the pronunciation data, so that in the case of the same sentence, the two 20 1294107 = the number of pictures of different lengths (the continuous lip picture of the teacher and the continuous lip picture of the learner) are correctly matched [such as the seventeenth Figure]. The teacher's lip video has m images, and the learner has n images [see Figure 18], so that the teacher's image is t(l), t(2),...5t(m) ), and the student’s image is

S(1), s(2), ···, <!〇, the purpose of dynamic time warping (DTW) is to find an optimal corresponding path on the plane formed by m, η. The starting point of this path is (丨,丨) The final point is (m,n), assuming d(i,j) is the distance between t(i) and s(j), that is, d(i,丨, the best path is also (1) , 1) to the minimum cumulative distance D(i,j) of (m,η). So rf(u) = and ―, and ―a)=幺ι(υ)—咕 is defined as follows: ί=1Μ ί (1) =< (i) ΚΙ) =< KU), Kl, 2), _.., (l, p), r(2, l)"."r(2,p)"." r(g,p) (2) t(l)Graph size=r(l)Graph size==ρΧί? (3) Calculate the starting position of d(l,l), and find it in sequence The cumulative distance of D(l, 2), D(l, 3)...

The following mathematical formulas are found: ^(i9 j) = min^ (4) D{Uj^\) + d{iJ) D(/-l,;-l) + 2rf(/,y) D(i- Lj) + d(i,j) After finding the cumulative distance, and then pushing back the minimum cumulative distance of the previous step, you can get the best path. Suppose the best path is: c(l), c(2),...c(p),c(k)=(t(k), s(k)), and the traceback algorithm for the best path is: 21 1294107 for m = k -1 to 1 i = c(m + 1) j = d(m + 1) next where m represents the number of times backtracked, and c(m) is used to record the position of the mth step i, d(m) is used to record the position of the mth step j, so every time you go back, you will find D(i,jl), D(il,jl), D(i-1,j) The direction of which cumulative distance value is the smallest is taken as the path of traceability. There are also some basic assumptions to speed up the DTW operation: :C(1)=(1,1), coffee)= (m,n) (5) .t(kl)<t(k) • (6) s( Kl)<s(k) • (7) s{k)-s{k -1) <1 • 1 t(k) - s{k) l< w, W is the window size (8) I . Boundary conditions Π .Incremental condition nr. Continuous condition IV. Window limitation V · Slope Constraint: After X times in the t direction, at least y times in the s direction. Therefore, in the sixteenth image, continuous images of the learners and learners using dynamic time warping (DTW) processing are also revealed, and similar images are removed by using dynamic time warping (DTW) to facilitate subsequent The outline of the graphic outline is scored on the implementation. The scoring mechanism using the graphical contour scale has some advantages, for example: it is calculated only for the contour of a certain graph, so 22 1294107

9 ο Ίχ c. i=0 The amount of calculation in the whole will be smaller than that of scanning the whole pattern, so the error will be relatively reduced, and the accuracy of the comparison of the comparison will be improved. In addition, since the ratio of the contour of the graphic is used as the method of scoring, there is no need to worry about the problem of normalization. Therefore, based on the above considerations, the present invention adopts a mechanism of graphical contour ratio as a criterion for scoring. At present, the pictures after Spatial-temporal [Time-Space Correspondence] are RGB full color (24bits), and the range of r, g, and B values is between 0 and 255, using the graphic outline of the educator and learner. The ratios are subtracted from each other, and the values obtained by subtracting the ratio of each teacher's and the learner's graphical outline (Equation 11) are summed (Equation 12) as follows:

Rate = W/Η

Rate is the method for calculating the aspect ratio of the image of the teacher and the student, η is the proportion of the outline of the figure of the ith piece of the teacher, 乂 is the proportion of the outline of the i-th picture of the learner, K is the number of pictures, and e is two The subtracted error value sums the average, and the current conversion score value, 0 to 100 points, is as follows:

MaxE = max(Ei), i = 1,···,η (11)

Score = 100-ΙΟΟχ—^— (12)

MaxE and MaxE is the maximum error value, 100*(E/MaxE) to get the error score, then use 100 to subtract the error score, then 23

1294107 What is the true score? (2) ii sensational lip and sound score: At present, due to the interference in the sound recognition, the interference from Yiwen to the external sound causes a big difference in the identification process, so it is regarded as X through the contour of the lips. The field scores the auxiliary parameters and compares the pronunciation corpus with the standard corpus to enhance the accuracy of the score. The invention utilizes the speech recognition technology and the speech scoring mechanism, and the mutual assistance of the 'missing visualized lip scores, and verifying that the combined sound is higher than the sound only using the sound or the lip to score higher on the sound signal processing, due to interception Most of the characteristics of the speech are firstly converted into digital signals by the speech input device, and the analog signals are pre-processed, and then the feature parameters are extracted to be used as the speech score. Comparison parameters. The flow and method of speech processing are further described below [see Figure 19]. (1) Digital signal conversion and sampling First, the digital signal conversion of the voice (S51) is performed by the voice input device of the computer as a analog signal, and then converted into a digital signal by the computer's voice input device, and then digital signal sampling (S52) 'And the sound box (S53), wherein the sound sampled data is 16KHz per second, and the sound box size is 512 points, about 32 milliseconds, and each sound box overlaps each other (0 verlap) 170 points, about one sound box One third of the. (2) Endpoint detection (S54) When a voice is recorded, there are usually a few paragraphs where the voice of 24 1294107 is muted. However, most of these silent parts are not needed. In order to obtain the required voice data more efficiently, endpoint detection is needed to remove the silent part of the voice head and tail. The present invention uses short-term energy (Short-Term Energy). ) and Zero Crossing and set a Threshold to determine if it is muted. ◎Short-time energy: In a sound, the sound energy emitted by the different time points will be different, so the energy in the silent part will be lower than that of the sound part, so just set The threshold value, when the volume exceeds the threshold value, it means that there is a voiced section. If it is not exceeded, it indicates the part that may be muted. In this way, you can know which voice frame is the required voice data. . The energy measurement is to add the average energy of all the sound energy values of a short time, as shown in the following formula: 1 N+m+l • Ek = N^lS(n)l (13) Represents the average of the energy of the kth frame, and N is the number of points sampled in a frame, and $(8) represents the nth point in the kth frame! 〇 is the starting point position of the frame. ◎ Zero-zero rate: The so-called zero-zero rate represents the number of vibrations of the sound signal passing through the zero point. The mathematical expression is as follows: Α = full Μ 心 1 heart (8)) - sgn_ -1))! (14) 25 (15) where (4)

In the equation, the 第gn(s(n)) represents the kth zero-zero rate, and if the nth point is greater than zero, then i is -1. However, the zero rate of the noise is greater than the zero rate of the air, and the zero rate of the air is greater than the zero rate of the sound. Therefore, if the air sound is detected, it must be detected by the zero rate and set. A threshold is used to judge. Therefore, the silent sound and the sound can be detected through the energy detection and the zero rate. (3) Feature parameter acquisition (S55)

When a parent speaks, he or she will have a specific pronunciation according to his or her gender, age, geography, etc. Even if the same person, the voice signals generated under different psychological conditions or physiological conditions will be different; Using the voice signal two waveforms to perform the comparison work, not only the processing capacity of the data is large, but also the recognition rate is very limited; therefore, when performing voice signal processing, it is necessary to obtain a more appropriate voice signal feature. And it is formed into a so-called feature vector. After processing, the original voice signal can be replaced by this feature vector as the basis for system identification. This process is the feature extraction of voice signals. Therefore, voice linear predictive marshalling (LPC) algorithms and cepstral parameters can be used in this process to manipulate feature parameters. ◎ Speech Linear Predictive Coding (LPC): 26 sample sounds are divided into _ _ and the calculation of the catch is also ί ~ financial effects ^ voice parameters voice samples can be. The theory of LPC is to assume that a prediction 'if the actual: z == linear combination, = county error reduced to the minimum LPC coefficient, such as ^ ^ Gui's predictor 'the coefficient is called (} ^ ak ' St ~M±Qzl/(n) ◎Cepstrum characteristics: (6) The average spectral coefficient of the energy speech signal in each frequency band. Although the == characteristic means that the scrambling of the speech signal is processed to be closer to humans. Hearing system, borrowing >n, ° w, but according to the results of the current research, the recognition rate κι Μ can often get higher speech-resolved speech = cepstrum feature has become the most commonly used (4) pattern Comparison score (S56) Seven methods using pattern matching, 24 points of speech and standard speech data are played by phonetic basis, and three to find the difference between the scored speech and the standard speech. The scoring speech is scored, and the minimum dynamic time warp (DTW) average error of the feature is obtained after the pattern matching is adjusted for the 强度 强度 intensity curve, the fundamental frequency trajectory, and the Mel cepstrum. The similarity between the voice and the standard voice High, the higher the score will be. 27 1294107 In addition, the acoustic model used in speech recognition is based on the Hidden Markov Model (HMM), which has been found in some previous studies. The hidden Markov model is basically a double stochastic process, and it is called hidden because there is a set of random processes that are hidden and invisible. In speech, it is like the state of human vocal organs in the process of vocalization. Changes are invisible, as changes in the throat, tongue, and mouth are not visible from observable sequences of arpeggio signals. Another set of stochastic processes is called an observation sequence, which is the probability of state observation. (state observation probab i 1 i ty ) to describe the probability distribution of various 5 eigen characteristic parameters observed in each state. The characteristics of HMM are just right for describing the characteristics of speech, and each state can be regarded as a channel ( Vocal tract) is in an articulatory configuration, while state observation probability is described in some The possibility of hearing various sounds in the utterance state. Through the above implementation description, the present invention has the following advantages: 1. The invention automatically determines the correctness of the lip shape when the sound is pronounced by the computer, and corrects the hairpin. The wrong position of the lip shape can make the invention an effective English scoring method. 2. The invention adopts the interactive pronunciation bridge teaching, so the learner can self-learn at home using the interactive teaching system of the invention. Bun, Dong T' and through the continuous picture of the lips, to strengthen the sound 28 1294107 section to improve the lip shape when the pronunciation is continued. 3. At the same time, the hair _ can improve the learner's interest by means of silk. Through the automatic scoring of the system, the accuracy of the lip type can be obtained, so as to effectively achieve the purpose of vocal correction. X曰, 4·Because listening to p is limited by hearing restrictions, it is difficult to make it difficult. Therefore, the present invention is effective in helping the hearing impaired to practice pronunciation by the continuous writing of the lips. Disabled: Review = Learn the correct language pronunciation. h ° faster 5. The present invention uses the correlation technique of lip alignment to evaluate the correctness of the pronunciation using the change of the pronunciation time profile, and makes the lip scoring mechanism = accurate estimation of the lip contour change when the driver is pronounced Increased. 6. The invention can check whether the pronunciation of the learner is correct through the interactive teaching, and can input the lip shape of the positive pronunciation, so that the learner can recognize that he is learning the language ^ It is lacking, so it can help the learner's effect in learning, and then the difficulty of learning. In summary, the embodiment of the present invention can achieve the expected use efficiency' and the specific structure disclosed therein, not only has not been seen in the same kind: in, and has not been disclosed before the application, Cheng has fully complied with the special According to the requirements of the Regulations 2, the application for invention patents is filed according to law, and the application for the patent is granted. 29 1294107 [Simple description of the diagram] The first diagram: the flow chart of the system of the present invention, the second diagram: the schematic diagram of the new English teaching sentence of the present invention, the third diagram: the learner's practice sentence selection diagram of the present invention Figure 4: Schematic diagram of the WebCam display of the present invention. FIG. 5 is a flow chart of the lip detection of the present invention. FIG. 6 is a schematic diagram showing the range of the detected lip block formed by the indentation of one-eighth of each of the left and right sides of the present invention. Figure 7: Lip image of the present invention after being cut by the RGB color system. FIG. 8 is a schematic view of the lip image of the present invention after being cut by the HSV color system. FIG. 9 is a result of the lip image of the present invention passing through the intermediate value filter. FIG. 11 is a schematic diagram of the four adjacent and eight neighboring blocks used in the numbering algorithm of the present invention. FIG. 11 is a view showing the original image of the present invention before the numbering algorithm. Invention of the image numbered after the numbering algorithm is not intended to be the thirteenth picture: the invention of the lip boundary detection diagram of the fourteenth: the capture of the present invention Lips Information Schematic Figure 15: Schematic diagram of continuous image of the instructor and learner after DWT processing of the present invention. Figure 16: Schematic diagram of the position corresponding to the present invention after passing through the DTW. Figure 17: After the invention is transmitted through the DTW Position corresponding to the mouth image 30 1294107 Schematic FIG. 18: Schematic diagram of the DTW matrix of the present invention FIG. 19: Flow chart of the speech processing of the present invention FIG. 10: Schematic diagram of the sound wave of the present invention after being detected by the endpoint [Main component symbol description]

<Invention> (S1) New sentence (S2) Select sentence (S3) WebCam couch (S4) Automatic lip interval selection (S41) Face range detection (S42) Lip color cutting (S43) Noise removal (S44) Lip block capture (11) Boundary (12) Boundary (13) Boundary (14) Boundary (S5) Scoring mechanism (S51) Digital signal conversion (S52) Digital signal sampling (S53) Frame (S54) Endpoint Detection (S55) Feature Parameter Acquisition (S56) Pattern Comparison Score (〇) Center Position (W) Width (H) (P) Pixel 31

Claims (1)

1294107 X. Patent application scope: 1. A method for assessing the pronunciation of audio and video on digital learning. The process steps include: (1) Adding a sentence, the teacher inputs the sentence to be added through the interface, and uses WebCam captures the lips of the instructor and extracts the images in the face into a picture of the lips, and saves the sentences and the captured lips into the database; (2) Selecting sentences and learning Select the sentence to be practiced from the database; (3) WebCam will take it, use WebCam to capture the lips of the learner, and capture the image in the picture into a picture of the lips; (4) Automated lip interval Select, through the color space conversion to cut the skin color of the face, and remove the noise, and the image is numerically calculated by algorithm to define the face range as the search for the lips boundary; (5) the scoring mechanism, The captured lip contour images and pronunciation corpus are compared with the standard lip images and standard corpus established in the database and scored. 2. The method for applying sound and video to digital learning as described in claim 1 of the patent application, wherein the system in the pronunciation evaluation method can list all the sentences stored in the database for The learner can easily select the sentence to be practiced. 3. The method for assessing the pronunciation and image applied to digital learning as described in the second paragraph of the patent application, wherein the sentences stored in the database are classified according to the order of the letters. 4. The method for assessing the pronunciation and image applied to digital learning as described in the first paragraph of the patent application, wherein the color conversion space used for face detection is selected in the automatic lip interval selection = 32 1294107 It is a YCbCr system. 5. The method for assessing sounds applied to digital learning as described in claim 4, wherein the face image is subjected to YCbCr conversion and the value of Cb is between [77, 130] and the value of Cr. Between [130, 173] to cut out the skin interval. 6. The method for assessing the sound and image applied to digital learning as described in claim 1, 4 or 5, wherein the upper, lower, left and right sides of the face are found by the cut skin region. Four boundaries, and the innermost movement of the leftmost boundary and the rightmost boundary, that is, the left and right indented one eighth is regarded as the critical line of the leftmost boundary and the rightmost boundary, if the searched range exceeds the critical boundary Then it is bounded by one-eighth of the indentation. 7. The method for assessing sounds and images applied to digital learning as described in claim 5, wherein the color system used for lip color cutting is an RGB color system, by the RGB color system. The R value, the G value, and the relationship between red and green are used to cut the image of the face range. The relationship is: 1 Ln RG 0 <- otherwise <U, where 'R represents the red value of the RGB color system, g represents the green value, and when the center is the minimum threshold value, it is the maximum upper limit of the threshold value, if the magnitude is between the G value and the G value The ratio is between and c/lim, then set to 1, otherwise it is 〇, forming a binarized image. 8. The method for assessing the pronunciation of sound and image applied to digital learning as described in item 7 of the patent scope, wherein the processed binarized image is further given 33 1294107 ==::r to the _ number, and then found out === Item = The applied sound and image are between 1.1 and 2.8 in digital. In the middle of the lip block, the range of the lip is in the range of ίο., as described in item 5 of the Uli 辄 第 第 音 与 与 与 与 , , , , , , , The color system used is the HSV color system, Η is expressed as hue value (Hue), S is not saturated, 豺 rainbow is illusive, and ν is expressed as “value” when converting the original lip image into HSV. Set the range of H, S, and V. When the value of each pixel is within the set range, set the prime point to 1, otherwise set it to 二 to form a binarized image. 11. The method for assessing sounds applied to digital learning as described in claim 10, wherein the range of values set is greater than 300 to less than 6 、, and the range of s is At [0· 16, 0·85], the value of V is between [〇· 21, 〇. 85]. 12. The method for assessing the sound and image applied to digital learning as described in the first paragraph of the patent application, wherein the HSV can be quantized into eight colors, namely: red, yellow, green, cyan, Blue, fuchsia, white and black. 13. The method for assessing the sound and image applied to digital learning as described in claim 5, wherein the lip color is combined with the RGB color system and the hsv color system to perform the lip block. In the RGB color system, the image of the face range is cut by the relationship between the R value and the G value of the rgB color system and the relationship between red and green. The relationship is: 1 ^ ^iim - ^lim [ lf \ G ~ 0 < — otherwise · where R represents the red value of the RGB color system, G represents the green value, where ^ is the minimum threshold, "the version is the maximum upper limit of the threshold, if the R value is between the G value The ratio is between ^ and ^, then set to 1, otherwise 0; and in the HSV color system, Η is expressed as hue value (Hue), S is expressed as saturation (Saturation), and V is expressed as brightness (Value) ), when converting the original lip image into HSV, set the range values of Η, S, and V. When the value of each pixel is within the set range, set the prime point to 1, otherwise set Cheng Yu, to form a binarized image. 14. If the scope of patent application is 13 The method for applying sound and image in digital learning, wherein the processed binarized image is given the same number by the numbering operation, and then the largest block is the lip. 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 16. The method for assessing sounds applied to digital learning as described in claim 13 wherein the range of values set is greater than 300 to less than 60, and the range of S is between [0.16, 0.85], and the value of V is between [0.21, 0.85]. 17. Apply the sound and image as described in item 13 of the patent application to the digit 35 1294107, and use the four neighbors as a block. Operation, and each point is judged one by one by means of recursive judgment, and the order of reciprocal judgment of the neighboring pixels of the same block is up, left, down, and right. When the recursive judgment ends, the representative has already marked a block. Finished and down Pixel blocks and search. 23. The method for assessing sounds and images applied to digital learning as described in claim 22, wherein searching for blocks and pixels is performed by searching for the number of blocks having the largest number of blocks. And on behalf of the lip block, the search method is to search from the upper left point of the image. If the first point is found to represent the uppermost position of the block, the coordinates of the point are recorded, and then the search for the lip contour is continued. The lower boundary, when the upper and lower boundaries are found, also finds the left and right borders of the lip contour, and then obtains the center position and width of the lip. The algorithm is: For i = 0 to the height of the image For j = 0 to the width of the image If pixel ( i,j)=The number with the largest number of blocks The top top = i < top ? i : top bottom = i > bottom ? i : top left = j < left ? j : left right = j > right ? j : right end If Next j Next i The top, bottom, left, and right represent the uppermost, lowermost, leftmost, and rightmost borders of the lip outline, respectively. 37 1294107 24. The method for assessing the pronunciation and image applied to digital learning as described in claim 1, wherein in the scoring mechanism step, a Dynamic Time Warping (DTW) algorithm is used. In conjunction with the Pattern Matching method, the similarity or difference between the lip image of the standard pronunciation teacher and the learner's mouth image is filtered to filter out the image that needs to be deleted or retained. The important key points are to make the scores different. If the similarity is higher, the score will be higher. ^ 25 · The application of sound and video in the digital learning method according to claim 24, wherein the operation of the _ time (four) can be between the sound of the test data and the sound of the reference material. Obtain a nonlinear response with minimum error; if the lip video of the reference data has m images, and the lip shape of the test data has n images, the lip video image of the reference data is t(l) 't(2),t(m) 'The lip image of the sound of the side material is s(l), S(2),...,S(n), dynamic time warp (10)) will be in the plane called n Find the best corresponding path on the line, the starting point of this impurity is (1, ^ the final point is (m, n) 'Assume d(i,]) is the distance between t(1) and s(j), that is, ^ (6)-Canb's best path is the most t accumulation of (1) to (4), f is from D(i,j); therefore, and W)丨' is defined as follows: I , < track "., ί (1,顺24),, ί(2, team is β =<_,_,..·,·), _",r(2,p) ” for the corpse) t(l)graphic size=!»〇) The size of the graph == 夕 ^ The difference between the starting position of d (l, l) and the following: column mathematics The sub-order finds D(1,2), d〇,3)....··= 38 1294107 , > 7 y / After finding this cumulative distance, 're-sequence _ minimum cumulative distance, you can get the most Good path; suppose the best ^^ C(l), C(2),...c(P),c(k)=(t(1),s(k)) i~p ' is the retrospective calculation of Du path The law is: D(i9 j) = miJ Dii -I J-1)+ 2d(i9 j) for m = k -1 to 1 i = c(m + 1) j = d(m + 1) [c(m), d(m)] = 3min< next Where m represents the number of times backtracked, c(m) is used to record the position of the mth step i, and d(m) is used to record the number! „The position of step j, so every time you go back to the back, you will find D(i,j-1) D(i —^卜1) D(i —L ]) 2 The cumulative distance value is the smallest in three directions. To be used as a traceback path. • As described in Patent Application No. 25, the method for assessing the pronunciation and image applied to digital learning, in which the boundary conditions can be used: ^,, gab (4); -1)^ί(Α:) · Continuous condition: 咻卜咻 κι · 视固限制:味卜啦), w is the window size; Shaanxi limit (S1〇Pe Constraint): walked in the t direction x - At least Germany must be in the 3rd side, 6th, 6th, a, J, J, X, and y, and the yoke and conditional acceleration of the dynamic time warping operation. 39 1294107 27·As stated in the scope of claim 1 Applying sound and image to the method of pronunciation evaluation in digital learning, in which the score of the graphic contour is used in the step of the scoring mechanism, and the ratio of the graphical contour of the reference data and the test data is subtracted, and each reference is used. The value of the data and the contour of the test data minus the value of the test data is added to total R Z(d) 17 _*=〇_ where Rate is the reference width and height ratio calculation method of the reference data and the test data '7; for the i-th image of the reference material, the fork is the ratio of the outline of the i-th image of the test data, κ is For all the number of pictures, E is the sum of the error values after subtraction, and the current conversion score, 0 to 100 points, is as follows: MaxE = max(Ei), i = 1,···,η Score = ί〇〇^100 X — MaxE and MaxE is the maximum error value, i〇〇*(E/MaxE) to get the error score, then use 100 minus the error score to get the true score. 28 ^ The method for assessing the pronunciation and image applied to digital learning as described in item 1 of the patent scope, wherein the speech signal processing in the scoring mechanism step must be performed before the feature parameters of the speech are taken out. The voice input device is used to input the spoken analog signal into the computer and converted into a number and a billion number, and the analog signal is pre-processed, and then the feature parameter is extracted to be used as a comparison parameter of the voice score.