KR20190123371A - Emotion recognition method and artificial intelligence learning method based on facial image - Google Patents

Abstract

The present invention relates to an emotion recognition method based on a facial image and an artificial intelligence learning method thereof. According to one embodiment of the present invention, the emotion recognition method based on a facial image comprises the steps of: acquiring and recognizing a plurality of facial images within a predetermined time; analyzing and tracking the facial expression of the facial image; making the analyzed and tracked facial image into data; and determining a final emotion state of the facial image, which is made into data. Therefore, according to one embodiment of the present invention, image data including a facial expression is acquired, thereby recognizing the emotion by analyzing the acquired image data.

Description

Emotion recognition method and artificial intelligence learning method based on facial image}

The present invention relates to an emotion recognition method and an artificial intelligence learning method based on facial images. More particularly, the present invention relates to an emotion recognition method and an artificial intelligence learning method based on facial images for acquiring image data including a facial expression and analyzing the acquired image data.

Multi-modal interface means a method of interface using voice, keyboard, pen, etc. for communication between human and machine. When multi-modal information through the multi-modal interface is input, a method of fusion and analysis of the input information in each of these modalities is used.

However, various learning algorithms can be used to extract and classify the features of the input information in such multi-modalities. Therefore, depending on which learning algorithm is used, the error rate recognized by analyzing it appears variously.

On the other hand, the prior art using a visual modality response with a high facial expression recognition rate has a limitation on the brightness when recognizing the facial expression of the object. In addition, there is a problem that the emotion recognition is not actually performed well compared to the invention that causes the natural emotion by measuring the object response by using the object response data of the artificial emotional state. And after extracting features using multi-modality, there was a problem that these features cannot be fused.

Therefore, in the technical field to which the present invention belongs, it is required to develop an invention that can detect various emotions expressed by an object by detecting a change in facial expression of an object, which is one of multi-modalities, and to solve such a problem (Patent Document 1). It is supplemented through), which will be described in detail with reference to the following.

(Patent Document 1) relates to an emotion recognition device using an expression of the face, an emotion recognition method and a recording medium thereof, and accordingly, face recognition for receiving the object information of the object in response to the presented stimulus to recognize the face of the object The emotions expressed by the object by recognizing the facial expression change of the object that changes in response to the stimulus based on the means and the face element extraction means for extracting the face elements of the object using the recognized face of the object and the extracted face elements of the object. Emotion recognition means for recognizing.

However, the aforementioned (Patent Document 1) has a problem in that the error range for recognizing facial expressions and emotions is large or recognized with other expressions or emotions.

Republic of Korea Patent Application Publication No. 10-2013-0015958

It is an object of the present invention to provide an emotion recognition method and an artificial intelligence learning method based on facial images for acquiring image data including a facial expression and analyzing the acquired image data.

In order to solve the above problems,

Emotion recognition method and artificial intelligence learning method based on facial image according to an embodiment of the present invention

a. Acquiring and recognizing a plurality of facial images within a preset time;

b. Analyzing and tracking the facial expression of the facial image;

c. Data of the analyzed and tracked facial images; And

d. Determining a final emotion state of the data facial image;

It may include.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, in step a, the facial image is first acquired, and the facial image is reacquired within a preset time from the time of acquiring the first facial image. can do.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the step b is to analyze the facial expressions of the plurality of facial images, respectively, and match the analyzed facial expressions to track the change in facial expressions facial image You can judge the feelings of the first time.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the first judged emotion is any one of seven emotions and 21 emotions divided into high, normal, low for each emotion It may include one.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, by analyzing the different facial images, if the change value of the eye and mouth is above a certain level, it is determined that the high, if a certain level, if the constant is low can do.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the seven emotions may include anger, aversion, fear, happiness, sadness, surprise, expressionless.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the step c may be data after indexing the facial expression change and the emotional state of the tracked and analyzed facial image.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the step d may compare the data of the facialized image and the previously stored data to derive the final emotional state.

In the artificial intelligence learning method based on the facial image derived the final emotional state according to an embodiment of the present invention,

a. Generating a pre-learning model and primary learning it;

b. Generating an initial learning model by performing secondary learning based on the pre-learning model;

c. Selecting data from the data learned in steps a and b based on the reliability of each label and determining the same;

d. Tertiary learning on the initial learning model;

e. Determining whether to converge in a predetermined direction by examining a learning model generated through the first to third learning; And

f. Generating a final learning model;

It may include.

Further, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the pre-learning model may include a function of extracting a vector of data.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the steps a and b may be further performed to improve the performance and reliability of the learning model.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, if the learning model trained by performing steps a and b has a lower performance or reliability than the pre-learning model, the learned data is deleted. And relearn primary and secondary learning.

Step c is an artificial intelligence learning method based on the face image to determine by determining only some of the labels of all the training data.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, if the reliability of each label is 0.9 or more, it can be determined as a learning model having high performance and reliability.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the d step may improve the performance of the model by performing active learning (Lctive Learing) to the initial learning model.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, the step e is to record the final data every time the first to third learning step is completed and the data converges in a predetermined direction A first convergence step of changing the environment of the learning data; And

A second convergence step of recording the final data of the first convergence step and storing the final data in the final learning model when the data converge in a predetermined direction;

It may further include.

In addition, in the emotion recognition method based on the facial image according to an embodiment of the present invention, if the final data is divergent in the first and second convergence step, deleting the data and regenerating the pre-learning model It may include.

Such solutions will become more apparent from the following detailed description of the invention based on the accompanying drawings.

Prior to this, the terms or words used in this specification and claims should not be interpreted in a conventional and dictionary sense, and the inventors may properly define the concepts of terms in order to best explain their invention in the best way possible. On the basis of the principle that it can be interpreted as meaning and concept corresponding to the technical idea of the present invention.

According to an embodiment of the present invention, the emotion may be recognized by acquiring image data including a facial expression and analyzing the acquired image data.

1 is an overall flowchart of a emotion recognition method based on a facial image according to an embodiment of the present invention.
Figure 2 is an exemplary use of the facial expressions of the facial expressions of the emotion recognition method based on the facial image according to an embodiment of the present invention.
3 is an overall flowchart of an artificial intelligence learning method based on facial images according to an embodiment of the present invention.

Specific aspects and specific technical features of the present invention will become more apparent from the following detailed description and embodiments related to the accompanying drawings. In the present specification, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are used to refer to the same components, even if displayed on different drawings. In addition, in describing an embodiment of the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only to distinguish the components from other components, and the nature, order, order, etc. of the components are not limited by the terms. If a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected or connected to that other component, but there may be another configuration between each component. It is to be understood that the elements may be "connected", "coupled" or "connected".

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

As shown in FIG. 1, an emotion recognition method based on a face image according to an embodiment of the present disclosure may include acquiring and recognizing a plurality of face images within a predetermined time (S110) and analyzing facial expressions of the face image. And tracking (S120), data analyzing and tracking the facial image (S130), and determining a final emotional state of the dataized facial image (S140).

First, the emotion recognition method based on the face image may acquire a plurality of face images by photographing the face with a photographing device such as a camera (S110).

In detail, S110 may first acquire a facial image through a photographing device, acquire a second facial image within a preset time based on the first acquisition time, and acquire a total of two facial images.

Here, the predetermined time is a time set in advance through the photographing device, it is possible to acquire a second facial image acquired secondly after a few seconds from the first facial image obtained first.

That is, the acquisition time of the primary face image and the secondary face image is acquired at an arbitrary time interval.

The primary facial image and the secondary facial image acquired in this way are matched with each other to analyze the facial expressions displayed on the facial image (S120).

Specifically, the facial expressions of the first facial image and the facial expressions of the second facial image are analyzed respectively, the analyzed facial expressions are matched to each other to track the change of expression, and the emotion of the facial image based on the tracked change value for the first time. To judge.

As shown in Figures 1 and 2, the emotions judged in the present invention judges a total of seven emotions, such as anger, disgust, fear, happiness, sadness, surprise, and expressionless, and by subdividing each emotion into one, A total of 21 emotions can be judged by judging emotions as high, normal or low.

Specifically, the change value of the facial expression of the first facial image and the facial expression of the second facial image, for example, by comparing and analyzing the change values of the tails and the tails of the eyes with different amounts of change according to each emotion, high, normal for each emotion It can be judged as low.

Here, the facial expression variation of the facial image detects the eye and mouth tails of the first and second facial images, respectively, and matches the eye and mouth tails of the primary and second facial images, respectively. If the change is more than a certain level, it is high.

That is, in the present invention, up to 21 emotions can be determined through at least two facial images.

After analyzing the facial expression through the facial image and determining the initial emotion, the facial image may be data (S130).

Specifically, the initial emotional state determined by a plurality of facial images and the change index of facial expressions of the facial image, for example, the index of eye and mouth changes, and after the initial emotional state is indexed, the final emotional state can be easily determined by data. To help.

The final emotional state of the photographed facial image may be determined by comparing the initial emotion state and the facial expression change value and the previously stored data.

Here, the pre-stored data is stored in a storage medium such as a server, the data provided from the outside, it can be a criterion for determining which emotion converged by matching the initial emotion state and the change value of the facial expression to the exponential data. have.

In addition, the externally stored data may include a BU3DFE database.

As shown in FIG. 3, the artificial intelligence learning method based on the facial image from which the final emotional state is derived may include generating a pre-learning model and first-learning it (S210) and a second-order based on the pre-learning model. Generating an initial learning model by learning (S220), Selecting the data based on the reliability of each label from the data subjected to the first learning step and the second learning step (S230) and the initial learning model Tertiary learning in step (S240), reviewing the learning model generated through the first to third learning to determine whether to converge in a predetermined direction (S250) and generating a final learning model ( S260).

The pre-learning model is a model generated based on data stored in a storage medium, for example, a server, which is provided from the outside, and the data provided externally in the present invention is data that can determine emotion through a face image, that is, a published BU3DFE database. Data provided by, but not limited to.

In addition, the pre-learning model is generated to be equipped with a function of extracting a vector of data, so that it is possible to extract the data of the derived data by repeatedly performing the data of the face image and the self-learning which derive the final emotional state. It is composed.

Based on the pre-learning model and the facial image from which the final emotional state is derived, primary learning is performed (S210).

Primary learning is to perform automatic learning based on the published BU3DFE database and previously stored facial images, and may be in the form of passive learning (or self learning) that performs learning based on given data.

In addition, the second learning may be performed based on the pre-learning model in which the first learning is completed, thereby generating an initial learning model (S220).

Secondary learning has the effect of improving the performance and reliability of the initial learning model by periodically repeating the self-learning and active learning based on the pre-learning model where the primary learning is completed.

Here, the performance of the learning model and the reliability of the data are improved by repeatedly performing the first and second learning.

For example, if the learning model before learning decreases in performance or decreases in reliability than the learning model repeatedly performing the first and second learning, the learning data for the first and second learning are deleted and the first learning is performed. Restarting the learning and secondary learning to improve the performance of the learning model and the reliability of the data (S225).

The initial learning model that performs the primary and secondary learning selects some data from the learned data (S230).

Specifically, among the initial learning models that improve reliability and performance by performing primary and secondary learning, data are selected by selecting the data separately based on the reliability of each label predicted while performing primary and secondary learning. Determine whether or not.

Here, the reliability criteria of each label may be selected in order of low reliability or in order of high reliability, and may be selected by selecting any one of them.

If the selected data has a reliability of 0.9 or more, for example, 90% or more, it can be judged as a learning model having high performance and reliability of the initial learning model, and then perform tertiary learning, and the reliability of the selected data is less than 0.9, for example, less than 90%. In operation S235, the performance of the learning model is rechecked by determining that the learning model has low performance and low reliability.

The initial learning model judged to have a reliability of 0.9 or more, for example, 90% or more, can perform the third learning to improve the performance and reliability of the learning model.

Specifically, the performance of the learning model may be improved by performing active learning based on data stored in advance in the initial learning model, which is determined to have a reliability of 90% or more, or a face image from which the final emotional state is derived.

In other words, by adding active learning to the initial learning model, the reliability can be improved from more than 90% to more than 95%, and up to 99% reliability.

The learning model that has completed learning by performing active learning further includes a first convergence step and a second convergence step of determining the accuracy of the learned learning model by checking each step of learning from the pre-learning model.

Specifically, in the first convergence step, when the completed data of the first learning step, the second learning step, and the third learning step, for example, the final completed data are selected and the data converges in a predetermined direction, Change the environment of the training data.

For example, in the first convergence step, after completing the first learning step, if the completed data is converged in a preset direction, the second learning step may reselect data to be learned or may be newly selected from the outside. By collecting data and performing secondary learning to learn various data, the performance and reliability of the learning model can be improved.

After performing the first convergence step, when the final data that has completed the first convergence step converges in a predetermined direction (second convergence step), the final learning model is designated and stored (S250).

In this case, when the final data obtained during the first convergence step and the second convergence step are diverged in a predetermined direction, the learned data is collectively deleted and the pre-learning model is regenerated to perform the first learning. It performs (S210).

In other words, whether the learning is conducted in the right direction by performing the first to third learning steps based on the data provided from the outside or data derived from the final emotional state, and the convergence step for each learning step, or the performance of the learning model. And it is possible to accurately determine the emotional state of the facial image by improving the reliability.

Although the present invention has been described in detail through an embodiment, this is for describing the present invention in detail, and the emotion recognition method and the AI learning method based on the facial image according to the present invention are not limited thereto. The terms "comprise", "comprise", or "have" described above mean that a corresponding component may be included unless specifically stated to the contrary, and therefore, excludes other components. Rather, it should be construed that it may further include other components, all terms including technical or scientific terms are to be generally understood by one of ordinary skill in the art unless otherwise defined Has the same meaning as

In addition, the above description is merely illustrative of the technical idea of the present invention, and those skilled in the art may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the one embodiment. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

Claims (17)

a. Acquiring and recognizing a plurality of facial images within a preset time;
b. Analyzing and tracking the facial expression of the facial image;
c. Data of the analyzed and tracked facial images; And
d. Determining a final emotion state of the data facial image;
Emotion recognition method based on facial image comprising a.
The method according to claim 1,
In the step a, the facial image is first acquired, and the facial image-based emotion recognition method reacquires the facial image within a preset time from the time of acquiring the first facial image.
The method according to claim 1,
Step b is a facial image based emotion recognition method for analyzing the facial expressions of a plurality of facial images, respectively, matching the analyzed facial expressions to track the change of facial expressions to determine the emotion of the facial image based on the facial image.
The method according to claim 3,
The first judged emotion is a emotion image recognition method based on the facial image, including one of seven emotions and 21 emotions divided into high, normal, low for each emotion.
The method according to claim 4,
A facial recognition method based on facial images that analyzes different facial images and judges that the change in eye and mouth is higher than a certain level, high if a certain level is normal, and lower if a certain level is lower.
The method according to claim 4,
The seven emotions are an emotion recognition method based on facial image including anger, disgust, fear, happiness, sadness, surprise, expressionless.
The method according to claim 1,
In the step c, the facial image based on the facial image is indexed after the change in facial expression and the emotional state of the tracked and analyzed facial data.
The method according to claim 1,
Step d is a emotion recognition method based on the facial image to derive a final feeling state by comparing the judged data and facial data stored in advance.
In the artificial intelligence learning method based on the facial image from which the final emotional state is derived,
a. Generating a pre-learning model and primary learning it;
b. Generating an initial learning model by performing secondary learning based on the pre-learning model;
c. Selecting data from the data learned in steps a and b based on the reliability of each label and determining the same;
d. Tertiary learning on the initial learning model;
e. Determining whether to converge in a predetermined direction by examining a learning model generated through the first to third learning; And
f. Generating a final learning model;
Artificial intelligence learning method based on facial image comprising a.
The method according to claim 9,
The pre-learning model is an artificial intelligence learning method based on facial image including a function of extracting a vector of data.
The method according to claim 9,
The a and b steps are repeatedly performed to improve the performance and reliability of the learning model further comprises a facial image-based artificial intelligence learning method.
The method according to claim 9,
When the learning model trained by performing steps a and b has lower performance or reliability than the pre-learning model, the artificial intelligence learning method based on face images that deletes the learned data and relearns the first and second learnings .
The method according to claim 9,
Step c is an artificial intelligence learning method based on the face image to determine by determining only some of the labels of all the training data.
The method according to claim 9,
When the reliability of each label is 0.9 or more, the artificial intelligence learning method based on facial image determined as a learning model having high performance and reliability.
The method according to claim 9,
Step d is an artificial intelligence learning method based on the face image to improve the performance of the model by performing active learning (Lctive Learing) to the initial learning model.
The method according to claim 9,
Step e is a first convergence step of recording the final data each time the first to third learning step is completed and changes the environment of the learning data when the data converge in a predetermined direction; And
A second convergence step of recording the final data of the first convergence step and storing the final data in the final learning model when the data converge in a predetermined direction;
Artificial intelligence learning method based on facial image further comprising.
The method according to claim 16,
When the final data is divergent in the first convergence step and the second convergence step, artificial intelligence learning method based on facial image comprising deleting the data and regenerating the pre-learning model.

Images