TW201028934A - Facial expression recognition method and system thereof - Google Patents

Abstract

A facial expression recognition method comprises an expression classifier training process, and an expression determining process. The expression classifier training process takes a plurality of optical flow chain codes of each image sequence as features, and adopts AdaBoost algorithm for training a plurality of strong classifiers, each strong classifier corresponds to one of a plurality of specific expressions. The expression determining process adopts the strong classifiers to perform classifier operation, and according to the result of classifier operation to perform expression recognition.

Description

201028934 VI. Description of the Invention: [Technical Field] The present invention relates to a facial expression recognition technology, and particularly relates to an optical flow vector using an image sequence as a feature, and is adapted to improve the fitness ( Adaptive Boosting (hereinafter referred to as AdaB〇ost) algorithm to train facial expression recognition method and system of expression classifier. [Prior Art] Automated analysis of human emotions can bring new milestones to human-machine interaction. Because quite a number of Duhui psychologists have shown that facial expressions are one of the most important forms of human communication; in recent years, more and more researchers have paid significant attention to this field, how to make machines Understand the way humans communicate non-verbally; for example, the development of facial expression recognition; is driven by huge research kinetic energy. b The number of studies on the field of expression recognition has been quite large over the years. The existing expression recognition methods can be mainly classified into three types: “based on feature-feat-based”, “model-based ((4) café (8)(4)”, and “by sample The "sample based" is briefly described below. The "feature-based" expression recognition method is based on the feature information of the face, for example, the integrity of a single image (glQbal) facial texture features, geometric features 3 201028934 The method of emotion identification is not like the "model-based" expression recognition method. Firstly, the expression model needs to be established first. Instead, learn a series of training samples and learn useful features to design and identify. The classifier of the ability, and then the expression recognition by the classifier: the invention is based on the "sample-based" expression recognition method for research and development. The research on the "sample-based" expression recognition method , P. Viola and M. Jones. ''Rapid object detection using a boosted cascade of simple features. Proc. of IEEE Int'l Conf. on Computer Vision and Pattern Recognition, Vol. 1. pp.511 -518. 2001. (hereinafter referred to as the document Viola) is to quickly find the matching features in the integral graph, and then use the AdaBoost algorithm Select the most suitable number of rectangular features to form a classifier with higher recognition ability, and finally save training and detection time in series. Sung Uk Jung, Do Hvoung Kim. Kwane Ho An. and Mvung Jin Chung. ^Efficient rectangle feature extraction for real-time facial expression recognition based on Adaboost·'' Proc. o疒/五五五/wf'/ Co»广o衫Intelligent Robots and Systems (IRQS-2005). pp 1941-1946. 2-6 Aug. 2005. (hereinafter referred to as the literature Chung) falsified the rectangular features proposed by the document Viola, providing a system for selection with a variety of trait types, and using the AdaBoost algorithm to make each expression Each of the five most suitable eigentypes is selected. Finally, the AdaBoost algorithm is used to find the position that best fits these features from the image, and combined with these features to form a classifier with high recognition. Expression. Literature Honebo Dene. Jianke Zhu, M.R. 201028934

Lvu. and I. King. ^Two-staee Multi-class AdaBoost for Facial Expression Recognition.Proc. of IEEE Infl Conf. on Neural Networks, dp.3005-3010. Orlando. Florida. USA. August 12-17,?_ 〇1 (hereinafter referred to as the document Deng) uses the Gabor filter to extract the wavelet coefficient features, and then uses the AdaBoost algorithm to select the most suitable features. The same feature is used to form a multi-class identification system for expression recognition. Incorporating the aforementioned literature and technology, the paper Viola is a model for using the AdaBoost algorithm to operate the expression recognition system in the past ten years. It is often modified by the method proposed by the document Viola; for example, the literature Chung increases the type of rectangular features' The dimension of the feature space gives the recognition effect a considerable degree of gain. As for the document Dene with the small glass coefficient as the feature 'can analyze the content of the expression sample in more detail, it also provides the basis for the high discriminating ability of the classifier. The invention is based on the fact that the expression is usually composed of a sequence of facial movement changes. A single static image often cannot fully express its meaning. Therefore, the optical flow vector of the image sequence is used as a feature, and the AdaBoost algorithm is used to design a classifier with discriminative ability for facial expression recognition. SUMMARY OF THE INVENTION Therefore, the object of the present invention is to provide a facial expression recognition method. Thus, the facial expression recognition method of the present invention comprises an expression classifier training program and an expression judgment program. The expression classifier training program is configured to train a plurality of strong classifiers according to a plurality of training images 201028934 image sequence samples, each strong classifier corresponding to one of a plurality of specific expressions, each training image sequence sample has been Marked as a special expression tag, the expression classifier training program includes the following steps: (a) obtaining a plurality of optical flow vector sequences according to each training image sequence sample, wherein each optical flow vector sequence corresponds to each facial image (b) for each optical flow vector sequence to obtain its red-light flow vector key code: (1) according to the same specific expression and the same face pixel position of the "quantity chain code" to obtain the corresponding specific The expression and the position of the aforementioned face pixel are one of the optical flow vector cores; (d) the needle is obliquely weak, and each of the four cores trains a weak, adaptable algorithm to perform the loop operation to the weak classifier. (10) A plurality of optimal formations for each-specific expression correspond to each-specific expression recognition, such as: _ performing a table group 兮 心 庆 · · 值 值 值 值 值 值 值 值 值 值If the combined confidence ranks, which is the specific expression, to judge the image sequence to be tested.

Another aspect of the invention is the system Φ ❹ . , 'p is providing a facial expression recognition system, the facial table-classifier establishing unit of the present invention, H includes 1 calculation calculation unit, and the feature calculation unit uses: an expression recognition unit. Isosaponins are used to obtain a sequence of images from a sequence of m images. 201028934 • Several sequences of optical flow vectors are obtained, and the corresponding optical flow vector chain codes are obtained for each sequence of optical flow vectors. The classifier establishing unit is configured to train a plurality of strong classifiers [each strong classifier corresponding to one of a plurality of specific expressions, wherein the classifier establishing unit comprises a weak classifier training module, - the best weak a classifier selection module' and a strong classifier combination module; the weak classifier training module is configured to obtain the corresponding specific expression according to the optical flow vector chain code belonging to the same specific expression and the same facial t position (4) And the front (4) part of the position of the line - line to the 4 core, and then for each - optical flow vector core training - weak classifier; the best weak classifier selection module is used to perform the loop operation using the adaptive improvement algorithm, Selecting a plurality of optimal weak classifiers for each particular expression from the weak classifiers; the strong classifier combination module is configured to linearly combine the best weak classifiers corresponding to the same specific expression The way forms a strong classifier that corresponds to each particular expression. The expression recognition unit includes the strong classifier trained by the classifier establishing unit, a confidence evaluation module, and an expression judgment module; the powerful classifier is configured to classify a sequence of images to be tested. Calculating a combined confidence value corresponding to each strong classifier; the confidence evaluation module is configured to obtain a corresponding plurality of confidence evaluation values according to the combined confidence values; Based on the confidence evaluation values, it is determined which particular expression the image sequence to be tested belongs to. The above and other technical contents, features, and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments. 7 201028934 Referring to Figure 1, a preferred embodiment of the facial expression recognition system of the present invention comprises an image pre-processing unit u, a feature computing unit 12, a classifier building unit 13' and an expression recognition unit 14. The classifier unit 13 includes a weak classifier training module 131, an optimal weak classifier selection module 132', and a strong classifier module 133. The expression recognition unit 14 includes a classifier operation module 141, a confidence evaluation module 142, and an expression determination module 143. The image pre-processing unit 11 is configured to pre-process a single image in the image sequence (training image sequence sample, image sequence to be tested). The feature calculating unit 12 is configured to obtain a plurality of optical flow vector sequences according to the image sequence, and obtain an optical flow vector chain code corresponding to each optical flow vector sequence. The classifier establishing unit 13 is configured to train a plurality of strong classifiers AU = 1, 2, ..., 7)' per-strong classifier & corresponding to a plurality of specific expressions. The expression recognition unit 14 is configured to determine a specific expression to which the image sequence belongs, and the classifier operation module 141 of the expression recognition unit A 14 has the strong points trained by the classifier establishing unit 13, wherein the specifics There are 7 kinds of expressions, which are a happy expression, an angry expression, a sad expression, a disgusted expression, a fear expression, a surprise, a feeling of too much - no expression. The invention has the usual knowledge in the field of implementation. An expression recognition is not limited to the above software. In the embodiment, the facial expression recognition system 1 of the present invention is included in the preferred embodiment of the facial expression recognition method of 201028934, and is determined by the expression recognition party of the present invention. program. In the figure, the virtual class training program, and the part of the two-arrow part of the expression class training program are used to describe the expression. The machine 'Figure 1 makes the solid arrow part corresponding to describe the expression. Refer to Figure 1 and Figure 2 for the data flow in the judgment program. The monthly training program includes the following steps, as shown in step 21,

^ + ^ , the private image processing step is because the face size of the training image sequence t image is not necessarily the same as the head direction (for example, the face symmetry direction). Therefore, in order to improve the accuracy of subsequent processing, by the image front The processing unit u cuts the face portion of each image in the training image sequence sample, and performs angle rotation correction on the cropped face portion (ie, 'make the face symmetry center line in the vertical direction) so that The facial features are aligned, and then the angle-corrected image is resampled (title (4)) to the same size (for example, '256 256 pixei). As shown in the feature calculation step 22, the feature calculation unit i2 first obtains the optical flow vector sequences according to the training image sequence samples, and then obtains the corresponding optical flow vector chain codes for each optical flow vector sequence. Each image sequence sample includes a plurality of face pixel positions, and each light flow vector sequence (and its corresponding optical flow vector chain code) corresponds to each face pixel position. In the preferred embodiment, the feature computing unit 12 uses the document Y. Bouquet, ^Pyramidal Implementation of the Lucas Kanade Feature__Tracker Description of the Al^orithm.^ Intel 9 201028934

Corporation Microprocessor Research Labs Φ The pyramid* structure method 'determines the optical flow vector sequence of each face pixel position in each image of the training image sequence sample, wherein each optical flow vector sequence has a plurality of optical flow vectors Information (Vjc, vj, each optical flow vector information (Vi, v,)) is represented by a tangent angle of 0 = arg(tan(%)). Since the training image sequence samples mostly change from no expression to A particular expression that changes significantly often requires several images, and different people's expressions change at different speeds. In order to normalize the subsequent processed optical flow vector chain codes to the same length for easy identification, the feature is calculated. The curve method (Quadratic) resamples the tangent angles of each optical flow vector sequence into a specific number of resampled positive knives i^resampkj'❷resamp丨e_2,...,0res〇n^e-m) Each of the resampled tangent angles is further quantized (in the preferred embodiment, <9__ is quantized to one of 8 tangent angle quantized values, and labeled as 12, 3, 4, 5 , 6, 7, or 8) to form the optical flow vector key code. As shown in step 23, the weak classifier training step is to establish a single π 13 weak classifier training module 131 by the classifier according to the same specific expression (each image sequence sample has been marked as a specific expression tag) and the same The optical flow vector chain code at the position of the face of the face is obtained by the optical flow vector core (Qptieal flQw _ns) corresponding to the specific expression at each facial position, and then the complex is trained with the optical flow vector core A weak classifier that corresponds to the position of each face pixel. In the preferred embodiment, the weak classifier training module 131 adds: the optical flow vector chain code of the same facial pixel position of the same-specific expression to each of the specific expressions - face light position 10 light 2010 28934

The stream vector core; then clustering is judged by a corresponding threshold value around each optical stream vector core, that is, any optical stream vector key code is at a distance from the European machine Reid of any of the optical stream vector cores. Within the threshold value, it is judged as a positive sample corresponding to a specific expression. Wherein, the threshold value is obtained by first sorting the distance of each optical flow vector chain code corresponding to the same facial pixel position to the core of the corresponding optical flow vector in each specific expression, and covering the distance by distance A specific ratio (for example, 80%) is used as a threshold for the threshold value, so the threshold value depends on the position of the facial element and the specific expression as shown in steps 24 to 25, and the optimal weak classifier selection step is for each specific The expression, by the optimal weak classifier selection module 132, performs an iterative loop operation using the AdaB〇°St algorithm to form a weak classifier pool (ροο1) formed from the weak classifiers (the weak classifier) The pool includes a plurality of weak classifiers corresponding to all possible face positions, and a plurality of (ie, 7) optimal weak classifiers and corresponding plurality of weight values are selected, wherein ^2, 2, . Γ, and each of the best weak classifiers (actually corresponding to the face of the face of the image. Since there are 7 kinds of expressions, so] 1 2 7 class combination step is the strong classifier combination module ...will form a weak (4)^ combination of the formula to form a pair (four)-specific The strong classification of love can be expressed by the following formula (1). 枷~) = Σα*Α·,, (from *)...........,=, .......... ...................... As described in equation (1), a specific 矣^mmstr (^ table 隋Q recognition depends on a strong knife cheek 15 and In the middle, Ct) is used to judge the optical flow of the face pixel position in the sample meaning (package chain). *-) can also be called "a specific table 11 201028934 two -: confidence value" if a certain combination of specific expressions of confidence value ^ sample read the specific expression of the closer the degree of closeness. Worth learning The AdaBoost algorithm used in the deconstruction steps 24 to 25 is a technique that is not described here. Referring to the circle 1 and FIG. 3, the expression judging program includes the following steps. The class 彳-M ^ is processed in step 21 ' with the image pre-processing 11 for each mother scene "cut, angle rotation correction" and re-sampling in the image sequence to be tested. As shown in step 32, the sign is calculated by the special ^^, ^ Λ, and the light flow corresponding to each of the best weak classifications is ## according to the image to be measured. Read the face of you. As shown in step 33, the prime position is the optical flow vector chain code obtained by the score 32, and the = computing module (4) uses the trained strong classification 2 to perform the classifier operation according to the steps. Corresponding to each strong classifier speech value. According to the confidence level evaluation module 142, the corresponding plurality of confidence evaluation papers are obtained by the confidence evaluation module 142. In the preferred embodiment, the combined confidence value of ":r is normalized by the following formula (1). To obtain the confidence evaluation value t.

Confidence evaluation value I Σ «*, < ❹ 2 As shown in step 35, the expression judgment module (4) is used: the degree evaluation value is extracted from which image-specific expression belongs, that is, in the confidence level In the evaluation paper, the largest value is selected, and the corresponding strong classifier, 12 201028934 'is a specific expression, that is, the specific situation to which the image sequence to be tested belongs is represented by the following formula (3). , arg [bearing % (confidence evaluation value).................................... The experiment used the Cohn-Kanade AU-Coded facial expression image sequence database to perform expression recognition. Among them, 317 image sequences were selected as the training image sequence sample group, and 158 was used as the test sample group. )ss_validatiQn) to increase the number of f-repository samples. In the experiment of the present invention, we temporarily set the number of weak classifiers r for each strong classifier to 2〇, and then for the specific 7 to be identified. The expression individually selects 2G best weak classifiers on the face pixel position. Therefore, on each test sample, a total of 140 face pixel positions (which may be repeated) of the optical flow vector chain code are used as features. Judging the expression attribute β of the test sample, if the value of Γ is changed in the experiment, various ❿ ^ (four) rates as shown in FIG. 4 can be obtained. In FIG. 4, the horizontal axis represents the π axis, and the average recognition rate obtained after verification is obtained. . @4 also shows two cases in which "were (indicated by dashed line)" and "not repeated (indicated by solid line)" have been selected in the process of selecting a weak classifier. We can find that the average recognition rate can reach 72% when the threshold value rises above about 8〇 in the case of “no repeat” selected Xiao classifier. In summary, the present invention adopts the optical flow vector chain code of the image sequence as the feature' and cooperates with the AdaBoost algorithm to design a strong classifier with a clock ability for the expression judgment of the image sequence, and can truly achieve the present invention. 13 The purpose of 201028934. The above is only the preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, that is, the simple equivalent changes and modifications made by the scope of the invention and the description of the invention are all It is still within the scope of the invention patent. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a preferred embodiment of the facial expression recognition system of the present invention; ❹ 疋/claw diagram illustrating the expression classification of the preferred embodiment of the facial expression recognition method of the present invention Figure 3 is a flow chart 'illustrating the expression judging program of the preferred embodiment of the facial expression recognition method of the present invention; and · I is thin for each: a graph showing the number and selection of different weak classifiers The various average recognition rates corresponding to the mode.

14 201028934 [Explanation of main component symbols] 1 .......... Facial expression recognition system 133... Strong classifier combination mode

System 11 .... image pre-processing unit 12 ... feature calculation unit 13 ... ... classifier establishment unit 131 .... Weak classifier training module 132...Best weak classifier selection module 14...Expression recognition unit 141....Classifier operation module 142 ..... .. confidence evaluation module 143 ... expression judgment module 21 ~ 25 · ... steps 31 ~ 35 .... steps

15

Claims (1)

201028934 VII. Patent application scope: 1. A facial expression recognition method, comprising an expression classifier training program, and an expression judgment program; wherein the expression classifier training program is used to train plural numbers according to a plurality of training image sequence samples A strong classifier, each strong classifier corresponding to one of a plurality of specific expressions, each training image sequence sample has been marked as a specific expression standard, and the expression classifier training program comprises the following steps: (a) obtaining a plurality of optical flow vector sequences according to each training image sequence sample, wherein each optical flow vector sequence corresponds to each facial pixel position; ' (b) is obtained for each optical flow vector sequence Corresponding to one of the optical flow vector key codes; ^ obtaining the optical flow vector corresponding to the specific expression and the aforementioned facial pixel position according to the optical flow vector chain codes belonging to the same specific expression and the same facial pixel position core; (d) training a weak classifier for each optical flow vector core; performing a cyclic operation using an adaptability improvement algorithm to select a plurality of weak classifiers corresponding to each specific expression from the = tool; And (f) forming the best weak classifiers in a linear combination to form a strong classifier for each specific expression; an expression recognition I: a second determination program for utilizing the strong classifier lines to include the expression determination program The following steps: 16 201028934 (g) Li Xiao's positive classifier calculates a first-level confidence value for each strong classifier for a to-be-measured sequencer; and ▲ (h) ϋ the combined confidence values The corresponding plurality of confidence levels "flat value α" determine which specific expression the image sequence to be tested belongs to. According to the facial expression recognition method described in claim 1, wherein in the step (1), each An optical flow vector sequence has a plurality of optical flow vector information 'per-light flow vector information is represented by a tangent angle. According to the facial expression (4) method of claim 2, wherein in the step (b), the tangent angles of each optical flow vector sequence are resampled into a specific number by a quadratic method. The resampled tangent angle 'each of the resampled tangent angles is further quantized into a plurality of tangent angle quantized values to form the optical flow vector key code 〇 4. According to claim 1 The facial expression recognition method, wherein 'the step (h) has the following sub-steps: (h 1 ) normalizing the special combination 心g heart value to obtain the confidence evaluation value; and (h- 2) The specific expression of the corresponding strong classifier in the median value of the confidence evaluation values is determined as the specific expression to which the image sequence to be tested belongs. 5. The facial expression recognition method according to the scope of the patent application, wherein the specific expressions are a happy expression, an angry expression, a sad expression, a disgusting expression, a fear expression, a surprise expression, and a Expressionless. 17 201028934 6. A facial expression recognition system, comprising: a feature-ten calculation unit for obtaining a plurality of optical flow vector sequences according to a training image sequence sample, and corresponding to each optical flow vector sequence One optical flow vector chain code; a classifier building unit, using w, station. 丨 early to train a plurality of strong classifiers, each strong classifier corresponding to a plurality of specific expressions - wherein the classifier The establishment unit includes a weak classifier training module, an optimal weak classifier selection module, and a strong classifier combination module, and the weak component _ training module is used according to the same specific expression and the same facial pixel & The optical flow vector key code of the device is used to obtain an optical flow vector core corresponding to the specific expression and the position of the aforementioned facial element, and then a weak classifier is trained for each optical flow vector core, the best The weak classifier selection module is configured to perform a loop operation by using an adaptability improvement algorithm to select a plurality of best weak classifiers for each specific expression from the weak classifiers. The module is configured to form the strong classifiers corresponding to each specific expression in a linear combination manner with the best weak classifiers corresponding to the same specific expression; and an expression recognition unit, including training by the classifier building unit The strong classifier, a confidence evaluation module, and an expression judgment module are used to perform a classifier operation on a sample sequence to be tested, and obtain a corresponding classifier for each strong classifier. a combination confidence value, the heart. The "evaluation module" is used to obtain a corresponding plurality of evaluation values based on the combined confidence values, and the expression judgment module is configured to perform evaluation according to the mentality The value is used to determine which specific expression the image sequence to be tested belongs to. 18 201028934 7. According to the facial expression recognition described in item 6 of the patent application scope, each optical flow vector sequence has a plurality of optical flow directions, unified information, and each optical flow vector information is represented by a tangent angle. . 8. The facial expression recognition system according to claim 7 of the claim, wherein the feature calculation unit uses the quadratic curve method to calculate the tangent angle of the ^^^ mother-optical flow vector sequence Resampling into a number of resampled = 1; each of the resampled tangent angles is then quantized to a complex (four) tangent angle quantized value where - to form the optical flow vector chain code. 9. The facial expression recognition system according to claim 6, wherein the confidence evaluation module normalizes the combined confidence values to obtain the confidence evaluation values. Η). The facial expression recognition system according to claim 6 of the patent application scope, wherein the expression determination module determines that the median value of the confidence evaluation value is the largest one of the corresponding strong classifiers It is the specific expression to which the image sequence to be tested belongs. '❿η.: According to the facial expression recognition system described in claim 6, the specific expressions are - happy expression, angry expression, sadness, a disgusted expression, a fear expression, a surprised expression , expressions. ..., 19