KR101204378B1 - A recognition system of facial expression for laughter therapy on mobile terminal, and a method thereof - Google Patents

Abstract

According to the present invention, after detecting a face in a camera image of a mobile terminal and then detecting a lip region, only the lip region is tracked in an input image frame; Normalize lips and lightening; The present invention relates to a smile expression recognition system and method for laughter treatment in a mobile terminal, characterized by recognizing a smile expression using a facial expression learning DB after extracting a feature vector.
According to the present invention, after detecting a face in a camera image of a mobile terminal and then detecting a lip region, only the lip region is tracked in an input image frame; Normalize lips and lightening; Recognize the expression of laughter using the facial expression learning DB after extracting the feature vector; The present invention relates to a smile expression recognition system and method for treating laughter in a mobile terminal, the method comprising measuring and displaying a smile amount corresponding to a recognized smile expression.

Description

A recognition system of facial expression for laughter therapy on mobile terminal, and a method

According to the present invention, after detecting a face in a camera image of a mobile terminal and then detecting a lip region, only the lip region is tracked in an input image frame; Normalize lips and lightening; The present invention relates to a smile expression recognition system and method for laughter treatment in a mobile terminal, characterized by recognizing a smile expression using a facial expression learning DB after extracting a feature vector.

According to the present invention, after detecting a face in a camera image of a mobile terminal and then detecting a lip region, only the lip region is tracked in an input image frame; Normalize lips and lightening; Recognize the expression of laughter using the facial expression learning DB after extracting the feature vector; The present invention relates to a smile expression recognition system and method for treating laughter in a mobile terminal, the method comprising measuring and displaying a smile amount corresponding to a recognized smile expression.

Recently, with the development of artificial intelligence and pattern recognition technology, facial expression recognition is one of the important technologies in the Human Computer Interface (HCI), and many researches on this are being conducted.

Considering the existing studies on facial expression recognition, the optical flow analysis method that recognizes facial expressions using facial muscle movement information through optical flow estimation according to facial expression changes, and local feature points (eyes, nose) Local expression method that recognizes the position of the mouth and compares it with the change of the feature part, and the whole face through Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) or Independent Component Analysis (ICA). Analysis includes holistic analysis method that statistically learns and recognizes face images.

However, when the conventional techniques are applied to these methods, a study was performed to recognize facial expressions using only one method under a general PC-class high performance facility.

In addition, in the stable environment with uniform ambient lighting, the researcher made facial expressions and recognized them, especially when the experimenter took a fixed posture.

Therefore, if you use a mobile terminal with a lower performance than the PC in all aspects, the performance cannot be effectively exhibited by using only one facial expression recognition method. Correction processing is required.

On the other hand, human facial expressions can be basically divided into six expressions: surprise, fear, sadness, anger, disgust and laughter (Ekman, P. (Ed). "Emotion in the human face (2nd ed.)"), In particular, the expression of laughter not only plays an important role in the interaction between people, but also improves the mental, physical and psychological health of the human being, and thus laughter treatment has a social and medical value.

However, in order to train laugh expressions or effectively treat unstable emotions through laughter, you must be able to recognize and measure your laughter expression on your device whenever you want, regardless of time and space. Using a mobile terminal equipped with a camera may be the best alternative.

However, as described above, it is difficult to immediately adopt the conventional technologies developed in the PC-class performance and stable environment to the mobile terminal.

In addition, the conventional technologies deal with not only the expression of laughter but also the entire facial expression, so it is difficult to apply to recognize and measure the expression of laughter for laughter training and laughter treatment.

Therefore, the facial expression recognition in real time in the mobile terminal, but the laughter facial recognition system and method specialized in recognizing the facial expressions are urgently needed.

In addition, not only the recognition of the expression of laughter but also the laughter expression recognition system and method that can perform the laughter therapy by itself by measuring the amount of laughter and presenting the result of laughter recognition is urgently needed.

Hereinafter, a description will be given of a conventionally developed facial expression recognition system or method.

First, the Republic of Korea Patent Publication No. 10-0874088 relates to a face recognition system using the image variance sensing, characterized in that to determine the image variance in the photographed image and apply the application corresponding thereto.

This is positive in that it tries to cope with various image variations, but it does not specialize in the expression of laughter and covers the whole expression, and also analyzes the feature points to perform the application that is most suitable for the image variation without correcting the image variation itself. It will be different from the problem consciousness of the present invention.

Secondly, Korean Patent Laid-Open Publication No. 10-2009-0050199 tracks motion information of feature points using an optical flow algorithm and recognizes facial expressions using a hidden Markov model.

In addition, in order to minimize the influence of lighting during the preprocessing, it is characterized by using the minimum minimum normalization method and the reference white.

This is positive in that it introduces a method for minimizing the influence of lighting, but it is not specialized in recognizing the expression of laughter in the mobile terminal and the correction of the necessary feature point is overlooked because of the unstable posture of the user. It is different from the problem consciousness.

Thirdly, Republic of Korea Patent Publication No. 10-0874167 relates to a facial expression correction system and method, when the feature points between the middle of the brow gathers in the center, when the feature points given to the inside of the eyebrows are lowered or at both ends of the lips It is characteristic to judge that the feature point has changed when the assigned feature points are lowered.

This is positive in that facial expressions are recognized using feature points, but the recognized facial expressions are related to frowning or depressive expressions, and are not specialized for recognizing laugh expressions in mobile terminals, and also correcting illumination and detected feature points. It is different from the problem consciousness of the present invention, such as not presenting.

Fourth, the Republic of Korea Patent Publication No. 10-2010-0066866 relates to a portable facial expression practice apparatus and method, characterized in that to practice the expression by detecting the region of interest in the face portion and by reinforcing the expression guide model therein. have.

This is positive in terms of using a portable device, but it is to practice the facial expression by judging the similarity between the detected facial pattern and the enhanced facial expression guide model. It also differs from the problem consciousness of the present invention, such as not presenting a correction process.

An object of the present invention is to detect only the lip area in the input image frame after detecting the face after detecting the face in the camera image of the mobile terminal; Normalize lips and lightening; The present invention provides a smile expression recognition system and method for the treatment of laughter in a mobile terminal, characterized by recognizing a smile expression using a facial expression learning DB after extracting a feature vector.

An object of the present invention is to detect only the lip area in the input image frame after detecting the face after detecting the face in the camera image of the mobile terminal; Normalize lips and lightening; Recognize the expression of laughter using the facial expression learning DB after extracting the feature vector; The present invention provides a smile expression recognition system and method for treating laughter in a mobile terminal, the method comprising measuring and displaying a smile amount corresponding to the recognized smile expression.

In the present invention, the laughter facial recognition system and method for the laughter treatment in the mobile terminal recognizes the laughter by tracking only the lip region in the frame after detecting the face and the lips in the image. There is a remarkable effect of recognizing facial expressions.

In the present invention, a smile expression recognition system and method for treating laughter in the mobile terminal can overcome the image deviation caused by the distance between the mobile terminal and the face by normalizing the lips, and also normalize the lighting by symmetric averaging and histogram matching. Overcoming the deviation of the image due to the unstable lighting environment in the outdoor, there is a remarkable effect that can accurately recognize the laugh expression.

In the present invention, the laughter facial recognition system and method for the laughter treatment can recognize the laughter whenever and wherever by using the mobile terminal by performing a graphic processing of the laughter facial recognition after the laughter facial recognition. There is a significant effect.

1 is a main configuration of a laugh expression recognition system for laugh laugh treatment in a mobile terminal according to an embodiment of the present invention.
2 is an example of a process of detecting a lip candidate region in a laugh expression recognition system for treating laughter in a mobile terminal according to an embodiment of the present invention.
3 is an example of a process of detecting a lip region in a smile expression recognition system for treating laughter in a mobile terminal according to an embodiment of the present invention.
4 is an example of a process of tracking the lips in a laugh expression recognition system for treating laughter in a mobile terminal according to an embodiment of the present invention.
5 is an example of a lighting normalization process in a laugh expression recognition system for treating laughter in a mobile terminal according to an embodiment of the present invention.
6 is an example of a method of recognizing a laugh expression in a laugh expression recognition system for treating laughter in a mobile terminal according to an embodiment of the present invention.
7 is an example of an image used in a facial expression learning DB in a laugh expression recognition system for treating laughter in a mobile terminal according to one embodiment of the present invention.
8 is an example of graphic processing for displaying the amount of laughter in a laugh expression recognition system for laughter treatment in a mobile terminal according to one embodiment of the present invention.
9 is a flowchart illustrating a laughter facial recognition method for laughter treatment in a mobile terminal according to an exemplary embodiment of the present invention.

Before describing the present invention with reference to the accompanying drawings, it should be noted that the present invention is not described or specifically described with respect to a known configuration that can be easily added by a person skilled in the art, Let the sound be revealed.

1 is a main configuration of a laugh expression recognition system for laugh laugh treatment in a mobile terminal according to an embodiment of the present invention.

The laugh expression recognition system 100 for treating laughter in a mobile terminal according to an embodiment of the present invention includes an image acquisition unit 110, a detection and tracking unit 120, a normalization processing unit 130, and a laugh expression recognition unit 140. ), Facial expression learning DB 150 and graphics processing unit 160.

First, the image acquisition unit 110 is a means for receiving an image containing a face and transmitting it to the detection and tracking unit 120, the detection and tracking unit 120 in real time to detect the image obtained from the camera embedded in the mobile terminal ) Function.

Preferably, the image acquisition unit 110 is designed to perform a function of transferring the image stored in the mobile terminal via the data cable or the image transmitted through other network means such as the Internet.

The detection and tracking unit 120 detects a face and a lip area, and detects only a lip area after the lip area is detected, and includes a face detection module, a lip detection module, and a lip tracking module.

The face detection module is a module that detects a face in an image transmitted from the image acquisition unit 110.

For example, the face detection module is designed to use the detection method using the Haar-like feature and the AdaBoost detection method together or to use only the AdaBoost method.

Preferably, when the Haar-like feature detection method and the AdaBoost detection method are used together, the method of applying the AdaBoost method after calculating the Haar-like facial feature value is designed.

Preferably, when the AdaBoost method is applied after calculating the Haar-like facial feature value, a classifier using the AdaBoost method is combined to generate a strong classifier and designed to combine and apply these strong classifiers in a cascade structure. do.

Preferably, the face detection module may be designed to employ other methods in addition to the above-described method according to the purpose of the present invention.

The lip detection module is a module that detects the lip area by using the face area detected by the face detection module.

The reason for detecting the lip region is to recognize the laughter expression using only the lip region after detecting the face once, rather than detecting the face for each input image frame.

That is, since the shape of the lips changes mainly when a person expresses a smile, the lip area alone can effectively recognize the smile, which can significantly reduce the computational load imposed on the recognition system and improve the speed.

An example in which the lip detection module detects a lip region will be described with reference to FIGS. 2 and 3.

2 is an example of a process of detecting a lip candidate region using a geometrical distribution feature of a face.

Since the lip region is at the bottom of the face, when the detected face region is divided into eight equal parts in the vertical direction, the lip region is set to be within 5/8 to 7/8 as shown in FIG.

Also divided into eight equally in the horizontal direction, it is set to have a lip region within 2/8 to 6/8 as shown in Fig. 2 (b).

Therefore, the lip candidate area is determined as an area within a range set in the vertical and horizontal directions, and includes the lip area in a rectangular shape as shown in FIG.

Preferably, the number divided into the vertical and horizontal directions to set the lip candidate region may be variously set, such as 10 equals or 16 equals.

Preferably, the vertical and horizontal ranges of the lip candidate region may also be variously set.

3 is an example of a process of detecting a lip region from a lip candidate region by using a distribution feature of gray pixels.

First, since the horizontal line in the middle of the lips has a darker feature than the surroundings, the center of the lips is detected using the contrast.

As shown in FIG. 3 (a), the sum of pixel values is calculated in the horizontal direction in the lip candidate area, and the lowest row becomes the lip center line, and the horizontal center position is the lip center in the lip center line.

,

라고 정의하면, 이 좌표가 입술 중심이 된다.That is, the horizontal center position and the corresponding vertical position

,

, This coordinate is the center of the lip.

After detecting the lip center, the lip area having the most effective range is detected from the center. For example, the lip area is experimentally designed according to the relationship between the lip center and the face area.

H/8,

H/8,

W/4,

W/4 로 설정하여 입술 영역을 검출한다.Specifically, if H is the vertical length of the detected face area and W is the horizontal length of the detected face area,

H / 8,

H / 8,

W / 4,

Set to W / 4 to detect the lip area.

Preferably, the numerical value of dividing the vertical and horizontal lengths of the face area may be variously set according to the purpose of the present invention.

Preferably, the lip detection module may be designed to employ other methods in addition to the above-described method according to the object of the present invention.

After the lip area is detected, the lip tracking module tracks the lip area without detecting the face and the lip in the image frame transmitted from the image acquisition unit 110.

This is particularly effective in capturing images using a camera of a mobile terminal. As the user grabs and captures the terminal with one hand, the motion of the image is inevitably severe for each frame.

An example of how the lip tracking module tracks the lip region will be described with reference to FIG. 4.

Basically, the sum of absolute difference (SAD) of pixel values between the candidate region of the current image frame and the lip region detected in the previous frame is obtained, and the position of the smallest value is used as the lip region of the current frame. Will be decided.

If the search area is a 15 x 15 area around the previous position, it can be expressed as follows.

는 후보 위치로서

이고,

는 이전 프레임에서 입술 영역의 픽셀값이고,

는 현재 프레임에서의 픽셀값이며,

는 이전 프레임에서 입술 영역의 시작 위치를 나타내고,

과

은 추적하는 입술 영역의 세로길이와 가로길이를 나타낸다.From here,

Is the candidate position

ego,

Is the pixel value of the lip region in the previous frame,

Is the pixel value in the current frame,

Indicates the starting position of the lips area in the previous frame,

and

Indicates the length and width of the tracked lip area.

However, if the calculation is performed at all candidate positions as described above, 15 x 15 x M x N subtraction operations are required and many operations must be performed.

Therefore, in the present invention, by applying a three-stage search method, a total of 27 x M x N operations are performed by calculating SADs at nine candidate positions in each step, which results in eight times the amount of computation than the method of calculating SADs at all candidate positions. Can be reduced.

Experimental review shows that there is no problem in the accuracy of tracking the lip area even if the computational amount is reduced by using the three-stage search method.

As shown in FIG. 4, in the first step of the three-stage search method, SAD calculation is performed at nine positions indicated as 1 and the smallest candidate position is selected.

In the second step, the SAD is calculated for 9 candidate positions centered on the candidate position where the SAD value was the smallest in the first stage, and the candidate position in the lower left is the smallest SAD value in the first stage. It is shown that the candidate position indicated by No. 2 is set and the SAD value is calculated.

In the third step, the SAD is calculated for 9 candidate positions centered on the candidate position where the SAD value was the smallest in the second stage, and the candidate position at the bottom center has the smallest SAD value in the second stage. Fig. 3 shows the setting of the candidate position indicated by 3 and calculating the SAD value.

As shown in FIG. 4, it can be seen that the distances for determining candidate positions for each step are reduced to 4, 2, and 1. FIG.

Preferably, the lip tracking module may be designed to adopt a different method in accordance with the object of the present invention in addition to the method described above.

After tracking the lip region in the image frame transmitted from the image acquisition unit 110, the normalization processing unit 130 normalizes the lips and the lighting.

The normalization processing unit 130 includes a lip normalization module and an illumination normalization module.

First, the lip normalization module is a module for normalizing the lip region and normalizes the size of the lip region.

This is particularly effective in capturing images using a camera of a mobile terminal. This is to compensate for the fact that the distance between the terminal and the face must be changed for each frame as the hand is held by the terminal.

In detail, when the distance between the terminal and the face increases, the size of the tracked lip region is inevitably reduced, and in the opposite case, the size is corrected.

For example, the size of the lip region is regularly normalized to a size of 64 × 32, and is preferably designed to be normalized to be equal to the size of the lip region stored in the facial expression learning DB 150.

After normalizing the size of the lip area, the lighting normalization module normalizes the lighting to exclude the effects of the lighting.

5 illustrates an example of a method in which the lighting normalization module normalizes the lighting.

In the original image as shown in FIG. 5 (a), normalized processing using normalized lighting normalization processing methods such as normalization, whitening, and histogram matching is indicated by (b), (c), and (d), respectively.

Normalization, whitening, and histogram matching methods globally correct the image, resulting in good lighting normalization for dark or light images.

However, these methods are analyzed that the lighting normalization effect is not large because the image is different by area when the light is emitted from the left or the right. Especially, when the sunlight is emitted from the side when the mobile terminal is held with one hand outdoors, We can't see the lighting normalization effect.

Therefore, in the present invention, in order to complement the conventional lighting normalization method, the normalization process is performed by combining the left-right symmetric averaging method and the histogram matching method using the fact that the human lip image is symmetrical.

Specifically, first, right and left symmetry averaging is performed on the detected lip regions, and local illumination distortion is improved, and next, histogram matching is performed on lip images with evenly distributed left and right contrast to improve lighting as a whole.

This is a method of performing full illumination normalization after performing partial illumination normalization, and it is possible to improve not only dark or bright images but also partially distorted lighting phenomenon.

As shown in (f) of FIG. 5, it can be seen that the lighting normalization effect is more stable than the conventional method even in an image in which the lighting change is severe.

After normalizing the lips and the lighting, the laugh expression recognition unit 140 recognizes the laugh expression and measures the amount of laughter.

The laugh expression recognition unit 140 may include a feature vector extraction module, a laugh expression recognition module, and a laugh amount measurement module, and may exclude the laugh amount measurement module according to design conditions.

The feature vector extraction module extracts the feature vector from the lip region and delivers the feature vector to the laugh expression recognition module.

For example, the feature vector extraction module is configured to extract the feature vector using a principal component analysis (PCA) method.

Preferably, depending on design conditions, a method other than the principal component analysis method may be employed.

The laugh expression recognition module recognizes the laugh expression using the feature vector and the facial expression learning DB 150 delivered from the feature vector extraction module.

FIG. 6 is an example of a method of recognizing a smile expression, and FIG. 7 is an example of an image used in a facial expression learning DB.

The laugh expression recognition module is a module that recognizes the laugh expression by classifying the feature vectors extracted by the principal component analysis through the expression learning DB 150. In FIG. 6, the laugh expression recognition using the three-layer backpropagation artificial neural network is performed. Is shown.

Preferably, according to the design conditions, the laugh expression recognition module may adopt a different classification scheme, of course.

The expression learning DB 150 is a database that detects and classifies a plurality of face images from the expressionless expression to the smiley expression through the detection and tracking unit 120, the normalization processing unit 130, and the laugh expression recognition unit 140.

Preferably, facial expression learning DB 150 is constructed in the same manner as used in the system of the present invention.

FIG. 7 illustrates a number of expressionless and laughter expressions applied for learning the facial expression learning DB 150.

According to the design conditions, the laugh amount measuring module is included in the laugh expression recognition unit 140.

The laughter measuring module is a module for measuring the laughter of the laughter facial expression recognized by the laughter facial recognition module, and compares the laughter of the laughter facial expression with the degree from the expressionless expression already learned in the facial expression learning DB 150 to the smiley facial expression. It is configured to measure the amount of laughter.

Alternatively, by using the size of the detected face and lips, the degree of laughter can be geometrically broadened to the feature vector, and the amount of laughter is defined as 0 to 100%, and then contrasted with the perceived smile vector. It may be designed.

FIG. 8 is an example of graphic processing for displaying a laugh amount, which is not displayed on the laugh amount bar in the case of no expression as in the case of (a), but on the laugh amount bar in the case of the smile expression as in the case of (b). It is displayed in real time.

The graphic processor 160 is an element that is connected to the detection and tracking unit 120, the normalization processing unit 130, and the laugh expression recognition unit 140 to display a graphic process of the detection process and the result.

According to the design conditions, the graphic processor 160 is configured to provide an interface through which the user can generally use the present invention by using a mobile phone terminal.

9 is a flowchart illustrating a laughter facial recognition method for laughter treatment in a mobile terminal according to an exemplary embodiment of the present invention.

It is to be noted that the description of the smile expression recognition system described with reference to FIGS. 1 to 8 is performed by the method described above.

Meanwhile, the above descriptions using only FIGS. 1 to 9 describe only the main matters of the present invention, and as long as various designs are possible within the technical scope of the present invention, the present invention may be applied to the configurations and methods of FIGS. 1 to 9. It is obvious that it is not limited.

110: image acquisition unit 120: detection and tracking unit
130: normalization processing unit 140: laugh expression recognition unit
150: expression learning DB 160: graphics processing unit

Claims (8)

As a laugh expression recognition system for treating laughter in a mobile terminal,
An image acquisition unit transferring an image input in real time from a terminal camera or an image stored in the terminal to a detection and tracking unit;
After detecting the face and detecting the lip area, only the lip area is tracked in the input image frame.
(a) Determining a search range of a preset range from an image frame input based on the lip region of the previous image frame,
(b) a detection and tracking unit for tracking only the lip region by calculating a position where the sum of absolute differences with the lip region pixel values of the previous image frame is the smallest at a candidate position having a predetermined interval within the search region;
A normalization processor for normalizing the size of the tracked lip region and normalizing illumination of an image;
A laugh expression recognition unit extracting a feature vector of a lip region from a normalized image and recognizing a laugh expression using a facial expression learning DB; And
And a facial expression learning DB for learning a plurality of facial expressions using the detection and tracking unit, the normalization processor, and the facial expression recognition unit. The method of claim 1,
The method of detecting a face by the detection and tracking unit is characterized in that using a Haar-like feature method and AdaBoost method, laugh expression recognition system for laugh treatment in a mobile terminal. The method according to claim 1 or 2,
The normalization processing unit uses a symmetric averaging method and a histogram matching method to normalize the illumination of the image, characterized in that laugh expression recognition system for the treatment of laughter in a mobile terminal. The method according to claim 1 or 2,
The laugh expression recognition unit further includes a laugh amount measurement module,
Laugh amount measuring module is characterized in that for measuring the laugh amount of the laugh expression, the laugh expression recognition system for the laugh treatment in the mobile terminal. An image acquisition step of transmitting an image input in real time from a terminal camera or an image stored in the terminal to a detection and tracking unit;
Detecting a lip area after detecting the face;
Track only the lips area in the incoming video frame
(a) Determining a search range of a preset range from an image frame input based on the lip region of the previous image frame,
(b) tracking only the lip region by calculating a position where the sum of absolute differences with the lip region pixel values of the previous image frame is the smallest at a candidate position having a predetermined interval within the search region;
A normalization processing step of normalizing the size of the lip region and normalizing illumination of the image;
Extracting feature vectors of the lip region from the normalized image; And
Recognizing a laugh expression using the extracted feature vector and the facial expression learning DB; laugh expression recognition method for the laugh treatment in a mobile terminal. The method of claim 5, wherein
In the step of detecting the face after detecting the face, the method of detecting the face is characterized by using a Haar-like feature method and the AdaBoost method, laugh expression recognition method for laughing treatment in a mobile terminal. The method according to claim 5 or 6,
In the normalization process of normalizing the size of the lip region and normalizing the illumination of the image,
In order to normalize the illumination of the image, the left and right symmetric averaging method and histogram matching method, characterized in that the laugh expression recognition method for the laugh treatment in the mobile terminal. The method according to claim 5 or 6,
After the step of recognizing the laugh expression,
And measuring the laughter amount of the recognized laughter expression.

Images