KR101697476B1 - Method for recognizing continuous emotion for robot by analyzing facial expressions, recording medium and device for performing the method - Google Patents

Abstract

A robot-based facial expression consecutive emotion recognition method comprises: constructing a database for storing images for each emotion; Calculating an independent variable through Principal Component Analysis (PCA) based on the distribution of images stored in the database; Learning the linear regression equation using the calculated independent variable; And applying the input image to the linear regression equation to calculate a value of emotion state. Accordingly, the robot continuously estimates the emotion of the human being through the expression of the human, so that the emotion can be shared between the robot and the human.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a robot-based continuous emotion recognition method, and a recording medium and apparatus for performing the same. 2. Description of the Related Art [0002]

The present invention relates to a robot-based continuous emotion recognition method, and a recording medium and apparatus for performing the same. More particularly, the present invention relates to a robot capable of continuously estimating emotion through human facial expression A facial expression based continuous emotion recognition method, and a recording medium and apparatus for performing the same.

Generally speaking, a robot is a robot that resembles a human and works on its own. But only a few decades ago, it was merely a fiction coming out of science fiction movies, and the actual robot was just an industrial machine with simple repetition. Then, as the computer technology developed in the modern world, researches on artificial intelligence have been actively carried out, and robots can also do more various things intelligently and actively.

Recently, the quantitative and qualitative development of daily life has enabled the robots to be accommodated not only in public institutions but also in the home. Therefore, an emotional robot that is familiar to the general public is required. For this purpose, a lot of research has been done not only on unidirectional robots that perform tasks determined by human commands, but also on robots that have emotion models, express emotions of robots, and recognize and respond to human emotions.

On the other hand, the emotional state of a human being is expressed through facial expressions, voices, and movements, and can also be estimated through the raw heart signals such as heart rate, blood pressure, and brain waves. Among these, the voice signal has a large influence of noise and is not independent of the speaker and the context, and the method using the bio-signal is difficult to be applied to the actual interaction between the robot and human due to the mounting problem of the equipment.

In addition, there have been many studies for recognizing the emotional state using facial expressions. However, most of them have been classified into types of facial expressions based on a single image information, and a continuous moving image information is also used to store a moving image And then based on this. However, when responding to the emotional state of the person in the actual interaction situation, if the state of the emotion is recognized with respect to the facial expression frame at that moment, there may be a failure of face tracking, an error of recognition, and the like.

  Wilhelm Wundt. &Quot; Grundriss der Psychologie [Outlines of psychology] ", Leipzig, Engelmann, 1896.   James A. Russell, Albert Mehrabian, "Distinguishing anger and anxiety in emotional response factors," Journal of Consulting and Clinical Psychology, 42, 1974, pp. 79-83.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and it is an object of the present invention to provide a robot-based continuous emotional recognition method based on a robot capable of continuously estimating emotions through human facial expressions will be.

Another object of the present invention is to provide a recording medium on which a computer program for performing a facial expression based continuous emotion recognition method of the robot is recorded.

It is still another object of the present invention to provide an apparatus for performing a facial expression based continuous emotion recognition method of the robot.

According to another aspect of the present invention, there is provided a method of recognizing consecutive emotion based on a facial expression of a robot, the method comprising: constructing a database storing images for each emotion; Calculating an independent variable through Principal Component Analysis (PCA) based on the distribution of images stored in the database; Learning the linear regression equation using the calculated independent variable; And applying the input image to the linear regression equation to calculate a value of emotion state.

In the embodiment of the present invention, the step of learning the linear regression equation may use an arousal value and a valence value corresponding to each emotion as a dependent variable, and the linear regression equation may be an arousal, - It can be a two-dimensional form of unevaluated axis (A axis) and pleasantness (Valence axis).

In an embodiment of the present invention, learning the linear regression equation may calculate a regression equation coefficient.

In an embodiment of the present invention, the step of calculating the emotion state value by applying the linear regression equation to the input image may include projecting the input image on an eigenvector obtained from the distribution of the database to calculate an independent variable step; And estimating an emotion state of the input image by substituting the independent variable into the linear regression equation.

In an embodiment of the present invention, the robot-based facial expression continuous emotion recognition method may further include a step of pre-processing images stored in the database before the step of calculating the independent variable through the PCA.

In an embodiment of the present invention, the step of preprocessing images stored in the database comprises: vectorizing the grayscale images of the images stored in the database; Normalizing the contrast of the vectorized images; Obtaining an average vector of normalized images; And obtaining a difference vector that is a deviation between the average vector and each image vector.

In an embodiment of the present invention, the step of calculating an independent variable through the PCA may include calculating an eigenvector by obtaining a covariance matrix through the difference vector; And projecting the difference vector to an eigenvector to calculate a reduced PCA value.

In an embodiment of the present invention, normalizing the brightness of the vectorized images may be performed through averaging and variance of each vector, and averaging and variance of the whole vector.

In the embodiment of the present invention, the step of constructing the database may include a step of numerically evaluating and storing a value of emotion state for each image by an evaluator.

In the embodiment of the present invention, the step of constructing the database may further include detecting a face region of each emotion-related image prior to the evaluation.

In the embodiment of the present invention, the step of applying the input image to the linear regression equation to calculate the emotion state value may include calculating a value of emotion state of each successive frame of the input image by applying a continuous emotion model, .

According to another aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for performing a facial expression-based continuous emotion recognition method of a robot.

According to another aspect of the present invention, there is provided an apparatus for recognizing a consciousness based on a robot, comprising: a database for storing images of each emotion; A first PCA unit for calculating an independent variable through PCA (Principal Component Analysis) based on the distribution of images stored in the database; A learning unit for learning a linear regression equation using the calculated independent variables; And a facial expression recognition unit for calculating an emotional state value by applying the input image to the linear regression equation.

In the embodiment of the present invention, the learning unit may use an arousal value and a valence value corresponding to each emotion as dependent variables, and the linear regression equation is an Arousal- Dimensional shape of the axis and the valence axis and the unpleasant axis (V axis), and the regression coefficient can be calculated by learning the linear regression equation.

In an embodiment of the present invention, the facial expression recognition unit may include a second PCA unit for projecting the input image to an eigenvector obtained from the distribution of the database to calculate an independent variable; And an emotion state output unit for substituting the independent variable into the linear regression equation to estimate the emotion state of the input image.

In an embodiment of the present invention, the facial expression recognition unit may further include a face region detection unit that detects a face region of the input image.

In an embodiment of the present invention, the robot-based facial expression sequential emotion recognition apparatus may further include a pre-processing unit for pre-processing images stored in the database before the PCA.

 In an embodiment of the present invention, the preprocessing unit includes: a vector unit for vectorizing gray-scale images of images stored in the database; A normalization unit for normalizing brightness and darkness of the vectorized images; An average part for obtaining an average vector of the normalized images; And a difference vector unit for obtaining a difference vector which is a deviation between the average vector and each image vector.

In the embodiment of the present invention, the database may store values of emotion states of respective images numerically evaluated by the evaluator.

In an embodiment of the present invention, the facial expression recognition unit may calculate a value of an emotion state of each successive frame of the input image.

According to such a robot-based facial expression sequential emotion recognition method, not using the PCA and the linear regression analysis but simply classifying the basic emotion types, the state value of each axis is numerically calculated The emotions of the subjects were reacted with more detailed conditions. Accordingly, since the robot continuously estimates the emotion through the human expression, emotion can be shared between the robot and the human being. Therefore, it can contribute to the development of the emotional robot and the household appliance field.

1 is a block diagram of a facial expression based continuous emotion recognition apparatus of a robot according to an embodiment of the present invention.
2 is a diagram showing an example of a main area of an AV plane stored in the database of FIG.
FIG. 3 is a chart showing the pre-evaluation values of emotion states stored in the database of FIG. 1; FIG.
4 is a detailed block diagram of the preprocessing unit of FIG.
5 is a diagram for explaining vectorization of a gray scale image performed by the preprocessing unit of FIG.
6 is a diagram showing an example of regression analysis modeling for facial expression recognition.
FIGS. 7 and 8 are flowcharts of a facial expression based continuous emotion recognition method of a robot according to an embodiment of the present invention.
9 is a graph showing the RMS error of V with respect to the size of the reduced dimension when the dimension is reduced through the PCA method.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a facial expression based continuous emotion recognition apparatus of a robot according to an embodiment of the present invention.

Recently, as the robot can be accommodated in the home, an emotional robot capable of interacting with the robot is required. Previous research on robots capable of these emotional interactions has limited the ability to recreate consecutive human emotions by dividing or classifying robot emotions into some basic emotions. Accordingly, the present invention proposes an emotion recognition apparatus in which the robot can continuously measure human emotion states based on a linear model from human facial expressions.

Referring to FIG. 1, a facial expression continuous emotion recognition apparatus 10 (hereinafter referred to as a device) of a robot according to the present invention includes an off-line unit 100 for collecting images in advance and learning a linear regression formula, And a facial expression recognition unit 300 for estimating the facial expression.

The apparatus 10 of the present invention can be implemented with software (application) for performing facial expression-based continuous emotion recognition of the robot, and the configuration of the off-line unit 100 and the facial expression recognition unit 300 can be implemented by the device Based continuous emotion recognition of the robot executed in the robot 10 of the present invention.

The device 10 may be a separate terminal or some module of the terminal. In addition, the configuration of the off-line unit 100 and the facial expression recognition unit 300 of the device 10 may be formed of an integrated module or may be composed of one or more modules. However, conversely, each configuration may be a separate module.

The device 10 may be mobile or stationary. The device 10 may be in the form of a server or an engine and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS) a wireless device, a handheld device, and the like.

The off-line unit 100 forms a linear model that can estimate a human emotion state before estimating the emotion of an image input in real time. The off-line unit 100 includes a database 110, a first PCA unit 150, and a learning unit 170. The off-line unit 100 may further include a preprocessor 130 for pre-processing image data input to the first PCA unit 150.

The database 110 stores images for each emotion.

Emotions can be defined as mental and physiological states associated with various emotions, thoughts, and behaviors, and subjective experiences are generally known to be related to mood, temperament, personality, and the like. Although there have been many studies to classify the general emotion of humans, it has been difficult to explain and classify human emotions because of the subjective and abstract nature of the emotions, and the opinions of many psychologists are different, Is not established.

In the present invention, emotional prototype model emotions which are mapped to the pleasant-displeasure, arousal-non-arousal axis are used for the sub-sentences suggested by James Russell. In general, the arousal-unaerobic axis is called Arousal and the pleasant-off axis is Valence. In other words, we modeled the basic Arousal-Valence (A-V) two-dimensional emotion and applied the emotion model to each axis. The emotion model is a method of expressing the state of emotion as a differential equation for time.

The emotion state stored in the database 110 is expressed in the form of arousal-non-arousal axis and pleasant-off-axis axis, that is, A-V two-dimensional, and expressed by the V value of emotion and the A value of emotion. For example, images of facial expressions corresponding to each emotion corresponding to each main value on the emotion plane can be collected and stored.

The database 110 collected in the present invention is used for two purposes. The first is to find the eigenvector that best represents the distribution of the database 110 in the PCA, and the second is to learn the regression equation through the obtained PCA value. Since the emotion to be recognized through facial expressions is based on the A-V 2-axis emotion model, each database can be divided into nine emotions corresponding to the main areas of the A-V plane and evenly collected. The main areas are neutral (0,0), pleasant (1,0), laugh (1,1), surprise (0,1), horror (-1,1), angry (-1,0), sadness -1, -1), sleepiness (0, -1), and comfort (1, -1).

In addition, the database 110 may store numerical values of emotion states of respective images evaluated by the evaluator. For example, a total of 90 face images are used for each of the images of the database 110 for each emotion, and one evaluator for each image sets the V value and A value, which are emotion state values, from 0 to 1 Quot; value " An example of evaluation values of the learning image and the emotion state is shown in Fig.

The face region may be extracted from the image before the images are stored in the database 110. [ To this end, the apparatus 10 may further include a face region detection unit (not shown) for detecting a face region from the images.

In facial analysis, the face is divided into the eye area and the mouth area. The human expression can be expressed by a combination of movement of the eye region such as eyebrows, eyelids, and eyes, and movement of the mouth region such as muscles around the mouth and mouth. Therefore, facial analysis is performed with reference to a specific region in which the facial expression change is pronounced. For example, the eye area is 11 to 90 in the horizontal direction and 50 to 21 in the vertical direction, and the mouth area is 21 to 21 in the horizontal direction. 80, and the length 71 to 100, respectively.

The preprocessing unit 130 preprocesses the images stored in the database 110 before performing the PCA in the first PCA unit 150.

Referring to FIG. 4, the preprocessing unit 130 includes a vector unit 131, a normalization unit 133, an averaging unit 135, and a difference vector unit 137.

The vector unit 131 vectorizes the gray scale images of the images stored in the database 110. Grayscale means that the image is displayed in black and white, and the value of each pixel is represented by one dimension. Grayscale images are displayed in an array of two dimensions in the horizontal and vertical directions, and images of 2x2 pixels can be vectorized as shown in FIG.

The normalization unit 133 normalizes the contrast of the vectorized images. Since the analysis using image information is vulnerable to the influence of illumination, it is necessary to normalize the brightness of the entire image. The vectorized full image can be normalized through the mean and variance of each vector, and the mean and variance of the whole vector, and can be performed by Equation 1 below.

Here, ptr is a vector to be normalized, and DB is an entire vector.

The averaging unit 135 obtains an average vector to highlight a difference between the vectors, and the difference vector unit 137 obtains a difference vector that is a deviation between the average vector and each image vector. This is the preprocessing step, and the first PCA unit 150 performs the PCA using this difference vector.

The first PCA unit 150 calculates an independent variable based on a PCA (Principal Component Analysis) based on the distribution of images stored in the database. PCA is a principal component analysis method that can solve low-dimensional problems by reducing the dimension of high-dimensional problems modeled as vectors.

The PCA process first finds eigenvectors in order of direction with large covariance for high dimensional data and then selects several eigenvectors in order of increasing eigenvalue. The PCA values can be obtained by projecting each difference vector in the database onto the selected eigenvectors. If the number of eigenvectors used is m, the value is reduced to m dimensions. The calculated PCA value is used as an independent variable to describe the characteristics of the facial expression in the linear regression analysis.

The learning unit 170 learns a linear regression equation using the independent variable calculated by the first PCA unit 150. [

는 독립변수,

는 종속변수이고,

는 회귀식의 계수,

는 잔차이다. Regression analysis is a statistical method for predicting the relationship of dependent variables to independent variables and analyzing the fitness, and establishing a mathematical relationship based on the data distribution of independent variables and dependent variables. Regression analysis is based on the relationship between given independent variables and dependent variable data, evaluates the fitness of two correlations, and predicts dependent variables for newly input independent variables. In the following equation (2)

Is an independent variable,

Is a dependent variable,

Is the coefficient of the regression equation,

Is the residual.

In the present invention, the PCA values of the eye region and the mouth region are used as independent variables for the two axes of emotion, and the values of Valence and Arousal, which are emotion states corresponding to the facial expressions, are used as dependent variables. Regression analysis modeling for facial expression recognition can be expressed as shown in FIG.

와

로 표현하였으며, 사전에 평가한 i번째 표정의 Valence 값과 Arousal의 값은 각각

와

로 표현하였다. Independent variables can be obtained by performing PCA analysis on the eye area and the mouth area of each image of the database 110. Here, the eye region and the input region value of the database i-th facial expression on the PCA k-th axis are represented by

Wow

The Valence and Arousal values of the i-th facial expression evaluated in advance are

Wow

Respectively.

를 찾을 수 있다. 이러한 방식으로 상기 학습부(170)는 회귀식을 학습한다. Independent variables can be calculated through PCA analysis of each image of the database 110, and the data set has the value of the evaluated dependent variable. Therefore, it is modeled as the determinant of FIG. 6, and the regression coefficient

Can be found. In this way, the learning unit 170 learns the regression equation.

The facial expression recognition unit 300 calculates the emotion state value in real time by applying the input image to the linear regression equation. When the linear regression equation is learned in the off-line unit 100, when a new expression is input, an independent variable is obtained by projecting the eigenvector obtained from the distribution of the database 110 to the linear regression equation, The state of emotion can be estimated.

1, the facial expression recognition unit 300 may include a second PCA unit 300 for projecting the input image onto an eigenvector obtained from the distribution of the database, that is, calculating an independent variable using PCA analysis, And an emotion state output unit 350 for substituting the independent variable into the linear regression equation to estimate the emotion state of the input image.

The facial expression recognition unit 300 may further include a facial region detection unit 310 for detecting a facial region of the input image before the PCA analysis. The input image may include a photograph, an image, and the like through a camera.

The facial expression recognition unit 300 can recognize the continuous emotion state by calculating the emotion state value of each successive frame of the input image.

As described above, in the present invention, the robot can continuously measure the emotional state of the human being based on the linear model from the facial expression of the human. When the facial expression of a human is inputted, the value of the emotion state is calculated based on a linear regression model, and the calculated emotion value is expressed in two coordinate axes of arousal and pleasantness / valence.

FIG. 7 is a flowchart of a facial expression based continuous emotion recognition method of a robot according to an embodiment of the present invention.

The robot-based facial expression continuous emotion recognition method according to the present embodiment can be performed in substantially the same configuration as the apparatus 10 of FIG. Therefore, the same constituent elements as those of the apparatus 10 of FIG. 1 are denoted by the same reference numerals, and repeated description is omitted. Further, the robot-based expression-based continuous emotion recognition method according to the present embodiment can be executed by software (application) for performing the robot-based expression-based continuous emotion recognition method.

Referring to FIG. 7, the robot-based consecutive emotion recognition based on the robot according to the present embodiment constructs a database for storing images for each emotion (step S10). The emotion state stored in the database is expressed in the form of arousal-non-arousal axis and pleasant-off-axis axis, that is, A-V two-dimensional, and expressed by the V value of emotion and the A value of emotion. For example, images of facial expressions corresponding to each emotion corresponding to each main value on the emotion plane can be collected and stored.

Prior to the evaluation, the step of constructing the database (step S10) may detect the face region of each emotion-based image and numerically evaluate and store the emotion state value for each image by the evaluator.

When the database is constructed, an independent variable is calculated through PCA (Principal Component Analysis) based on the distribution of images stored in the database (step S30).

A preprocessing process may be performed before the step of calculating the independent variable through the PCA (step S30).

Referring to FIG. 8, the preprocessing step includes vectorizing the gray-scale images of the images stored in the database (step S21), normalizing the brightness of the vectorized images (step S22), calculating the average vector of the normalized images (Step S23) and obtaining a difference vector which is a deviation between the average vector and each image vector (step S24). The step of normalizing the brightness and darkness of the vectorized images (step S22) may be performed through averaging and variance of each vector, and averaging and variance of the whole vector.

The step of calculating the independent variable through the PCA (step S30) includes calculating an eigenvector by obtaining a covariance matrix through the difference vector obtained in the preprocessing step, projecting the difference vector to an eigenvector, .

Then, the linear regression equation is learned using the calculated PCA value as an independent variable (step S50). In the step of learning the linear regression equation (step S50), an arousal value and a valence value corresponding to each emotion can be used as dependent variables. In this case, the linear regression equation is a two-dimensional shape of an Arousal-non-arousal axis (A axis) and a pleasantness (Valence axis). The regression coefficient is calculated through the linear regression learning.

When the regression formula learning is completed, the emotion state can be estimated from the image input in real time. To this end, the emotion state value is calculated by applying the input image to the linear regression equation (step S70). In this case, a continuous emotion state can be estimated by applying a continuous emotion model to calculate values of emotion states of successive frames of the input image.

Specifically, the input image is projected onto an eigenvector obtained from the distribution of the database, that is, an independent variable is calculated using PCA analysis, and the emotion state of the input image is calculated by substituting the independent variable into the linear regression equation . The input image may include a photograph, an image, and the like through a camera. Before the PCA analysis, the face region of the input image may be detected, and only the detected face region may be PCA analyzed.

As described above, in the present invention, the robot can continuously measure the emotional state of the human being based on the linear model from the facial expression of the human. When the human facial expression is inputted, the emotion state value is calculated based on the Liner regression model, and the calculated emotion value is calculated based on two coordinate axes of arousal-non-arousal (A axis) and pleasant- Lt; / RTI >

Hereinafter, the effect of the robot based on the expression-based continuous emotion recognition method according to the present invention will be verified.

Emotion is a very abstract concept, which makes it difficult to define the state numerically, so it is difficult to accurately assess the subject 's emotional state. It is also difficult to determine the authenticity of the identified system parameters when assuming a continuous emotional system.

In the present invention, the accuracy of the recognition of the emotion state quantified by comparing the facial expression analysis result with the value of the emotion state evaluated in advance was evaluated. The continuous emotional state estimation showed the response characteristics of the continuous emotional model according to the change of the emotional state with respect to time based on the results of the actual facial expression analysis, and analyzed the results.

9 is a graph showing the RMS error of V with respect to the size of the reduced dimension when the dimension is reduced through the PCA method.

The facial expression database for constructing the facial expression analysis system used nine facial expressions for each person, with a total of 90 images, 10 for each facial expression. The analytical method is to project each of the facial expressions as m data on m eigenvectors that best describe the distribution of data and to represent the data in m dimensions. The m dimension values of the eye area and the mouth area are used as independent variables, The linear regression equations are learned by using the state values as dependent variables.

For the experiment, PCA analysis was performed on 89 images except for one image used in the test, and the linear regression equations were learned. The errors of the V and A axes were calculated for all the images. The value of the error is the relative size of the emotion dimension when the maximum value of each emotion dimension is 1.

FIG. 9 shows a root-mean-square (RMS) error of V with respect to the size of the reduced dimension when the dimension is reduced through the PCA method. As the number of eigenvectors used as independent variables increases in the linear regression equation, the accuracy tends to increase gradually. The mean RMS (Root-Mean-Square) error of Valence and Arousal is 0.3335, which is an excellent estimation performance.

The present invention uses PCA analysis and linear regression analysis to classify emotional states through expression, rather than simply classifying the basic emotion types, but numerically calculates the state values of each axis for the AV two-dimensional emotion model , And can respond to the subject's emotion with more granular conditions.

In addition, we propose a method to estimate the state of consecutive emotion over time by introducing a continuous emotion model in order to apply real - time emotion recognition and real - time interaction to confirm emotional state at necessary moment.

Such a robot-based facial expression sequential emotion recognition method may be implemented in an application or in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be ones that are specially designed and configured for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims. You will understand.

10: Robot's Facial Expression Continuous Emotion Recognition Device
100: Offline part 110: Database
130: preprocessing unit 150: first PCA unit
170: learning unit 300: facial expression recognition unit
310: face area detecting unit 330: second PCA unit
350: emotion state output unit 131: vector unit
133: normalization part 135: average part
137: car vector part

Claims (12)

An emotional state discrimination method using an AV two-axis emotion model based on A axis (Arousal - non-awake axis) and V axis (Valence - unpleasant axis)
The evaluator evaluates the emotion state evaluated by the evaluator as the A value corresponding to the A-axis and the V-value corresponding to the V-axis by the evaluator with respect to the emotion state according to the facial expression of the images of each emotion and the facial region of the emotion- Building a database for storing the data;
Preprocessing images for each emotion stored in the database;
Calculating an independent variable through PCA (Principal Component Analysis) based on the distribution of the pre-processed images for each emotion;
Learning the linear regression equation using the calculated independent variable; And
Applying an input image to the linear regression equation to calculate a value of an emotion state,
The pre-
Vectorizing the gray-scale images of the eye region and the input region provided in the face region of the emotional images;
Obtaining an average vector of the eye region and an average vector of the input region; And
Calculating a plurality of eye area difference vectors that are differences between the average vector of the eye area and each of the eye areas and a plurality of mouth area difference vectors that are differences between vectors of the mean and the input areas of the mouth area, and,
Wherein the step of calculating the independent variable comprises:
Calculating a PCA value for the eye area through the PCA using the plurality of eye area difference vectors and calculating a PCA value for the mouth area through the PCA using the plurality of eye area difference vectors; and,
The step of learning the linear regression equation includes:
Constructing a data set having the PCA value for the eye region and the PCA value for the input region as independent variables and the A value and the V value as dependent variables;
Calculating a regression coefficient by linear regression analysis of the data set; And
And learning the linear regression equation including the regression coefficient,
Wherein the step of calculating the emotion state value by applying the linear regression equation to the input image comprises:
Projecting the input image onto an eigenvector obtained from a distribution of the database to calculate an independent variable; And
And substituting the independent variable into the linear regression equation to estimate the emotion state of the input image. The method according to claim 1,
Wherein the linear regression equation is a two-dimensional form of arousal-non-arousal axis (A axis) and pleasantness (Valence) -incomfort axis (V axis). The method according to claim 1,
The step of pre-processing images stored in the database comprises:
Further comprising the step of: normalizing the brightness of the vectorized images. The method according to claim 1,
The step of calculating the independent variable via the PCA comprises:
Calculating an eigenvector for an eye region by obtaining a covariance matrix through a difference vector of the eye region, calculating an eigenvector for the input region by obtaining a covariance matrix through a difference vector of the input region, And
Calculating a PCA value of a reduced-size eye region by projecting the difference vector of the eye region on an eigenvector for the eye region, projecting the difference vector of the input region onto an eigenvector for the input region, And calculating a PCA value of the input region of the robot. 4. The method of claim 3, wherein normalizing the intensity of the vectorized images comprises:
A robot based facial expression consecutive emotion recognition method, which is performed through averaging and variance of each vector, and averaging and variance of whole vectors. 2. The method of claim 1, wherein the calculating the emotion state value by applying the input image to the linear regression equation comprises:
Further comprising applying a continuous emotion model to calculate a value of an emotion state of each successive frame of the input image. A computer-readable recording medium on which a computer-readable recording medium for performing a facial expression based continuous emotion recognition method according to any one of claims 1 to 6. An emotional state discrimination device using an AV two-axis emotion model based on A axis (Arousal - non-awake axis) and V axis (Valence - off-axis axis)
The evaluator evaluates the emotion state evaluated by the evaluator as the A value corresponding to the A-axis and the V-value corresponding to the V-axis by the evaluator with respect to the emotion state according to the facial expression of the images of each emotion and the facial region of the emotion- A first PCA unit for calculating an independent variable through PCA (Principal Component Analysis) based on the distribution of images stored in the database, and a second PCA unit for calculating an independent variable using a linear regression equation An offline part including a learning part for learning the learning part; And
And a facial expression recognition unit for calculating an emotional state value by applying the input image to the linear regression equation,
The off-
A preprocessor for preprocessing images of each emotion stored in the database before the PCA; Further comprising:
The pre-
A vector unit for vectorizing the gray-scale images of the eye region and the input region included in the face region of the emotional images, an averaging unit for obtaining an average vector of the eye region and an average vector of the input region, And a difference vector section for calculating a difference in eye area difference vector which is a deviation between a vector and a vector of each of the eye areas and a difference between a mean vector of the mouth area and a vector of each of the mouth areas,
The first PCA unit,
Calculating a PCA value for the input area through the PCA using the PCA value for the eye area and a plurality of the input area difference vectors through the PCA using the plurality of eye area difference vectors,
Wherein,
A PCA value for the eye region and a PCA value for the input region are set as independent variables, and a data set having the A value and the V value as dependent variables is constructed, and the data set is subjected to a linear regression analysis Calculating the expression coefficient, learning the linear regression equation including the regression coefficient,
The facial expression recognition unit
A second PCA unit for projecting the input image to an eigenvector obtained from a distribution of the database to calculate an independent variable; And
And an emotion state output unit for substituting the independent variable into the linear regression equation to estimate the emotion state of the input image. 9. The method of claim 8,
Wherein the linear regression equation is a two-dimensional shape of arousal-non-arousal axis (A axis) and pleasantness (Valence) -incomfort axis (V axis). 9. The apparatus according to claim 8,
Further comprising a face region detection unit for detecting a face region of the input image. 9. The method of claim 8,
Wherein the preprocessing unit further includes a normalization unit for normalizing the lightness and darkness of the vectorized images. 9. The apparatus according to claim 8,
And calculates a value of an emotion state of each successive frame of the input image.

Images