KR102054058B1 - Analysis method of relations of face movements and the system thereof - Google Patents

Abstract

The present invention relates to a facial characteristic analysis method and a system using a facial motion relation that performs face analysis based on the relation between movements in a facial region. The present invention relates to a dynamic analysis of a facial microregion based on a relationship analysis deep network. By analyzing the relationship between features, face dynamic analysis such as gender recognition, age estimation, biometric authentication and facial expression recognition can be analyzed.

Description

Facial Characteristic Analysis Method Using Facial Motion Relationship and Its System {ANALYSIS METHOD OF RELATIONS OF FACE MOVEMENTS AND THE SYSTEM THEREOF}

The present invention relates to a method and system for analyzing facial characteristics using a facial motion relationship, and more particularly, to a technique for performing facial analysis based on a relationship between motions in a facial region.

Face analysis technology has emerged as an important technology for self-identification through identity information extraction, human-computer interaction through facial expression recognition, and personalized service based on gender and age estimation. Facial movements are known to provide important clues for face analysis in cognitive and psychological studies, and research has been reported to be useful in facial authentication-based facial motion analysis studies for biometric authentication and age estimation.

Recently, as the convolutional neural network (CNN) has been successfully applied in various fields such as image recognition, behavior analysis, and medical imaging, various research and technology results based on deep learning have been published. Various research results were also published regarding facial dynamic analysis, mainly using CNN in three dimensions or using LTM (Long Short Term Memory), a kind of CNN and purified neural network. However, researches and techniques on the relationship between the movements of the facial microregions are insufficient.

Recent research has shown that the dynamic information that occurs when a particular face is made is also useful in gender recognition. This is based on the cognitive and psychological studies that when men and women make expressions, men tend to hide expressions than women, and women are relatively accustomed to interactions through expressions. However, the existing research or method has disclosed a technique for extracting a dynamic feature according to the movement in the partial region of the face and gender recognition, but there is a limit not to consider the relationship between the movement of each region.

Accordingly, the following embodiments propose a relationship-based new deep network technology based on the observation that there is a relationship between movements in each face area when a person makes an expression.

The following embodiments provide a method and system for analyzing facial characteristics using facial motion relations using the facial feature relations.

Specifically, the following embodiments are a method for analyzing facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc. by analyzing the relationship between the dynamic features of the facial microregions based on the relationship analysis deep network. And to provide a system.

According to an aspect of the present invention, there is provided a method of analyzing facial characteristics using a facial motion relationship. And analyzing the relationship between the object features including the dynamic features of each region of the face by using a deep network to interpret the face dynamic analysis.

The encoding of the dynamic feature for each region may include extracting a facial feature map using a convolutional neural network (CNN) for each frame of the sequence of the face gesture image, and extracting the facial feature map. Classifying the partial region to extract the dynamic feature between two objects based on a long short term memory (LSTM) network, and adaptively encoding the importance of the relationship between the two objects using the dynamic feature. can do.

The extracting of the facial feature map may be used as the convolutional neural network by transfer-learning parameters of a plurality of convolutional layers according to an image model trained with a large amount of face data.

The extracting of the dynamic feature may classify the extracted facial feature map into a plurality of partial regions of the face and use the input as the input of the LSTM network, and extract the dynamic feature from the LSTM network.

The object feature may be a fusion of the dynamic feature and a location feature of the corresponding object.

The analyzing of the face dynamic analysis may include analyzing the relationship between the object features of the face gesture image based on the deep network and analyzing the face dynamics such as gender recognition, age estimation, biometric authentication, facial expression recognition, and the like. Can be interpreted.

According to an embodiment of the present invention, a facial characteristic analysis system using a facial motion relationship may include an image acquirer for acquiring a face gesture image of a user, and encoding dynamic characteristics for each region in a partial region of a face according to face movement from the face gesture image. The feature encoding unit and the relationship analysis unit for analyzing the face dynamic analysis by analyzing the relationship between the object features including the dynamic feature for each region of the face using a deep network.

The feature encoding unit extracts a facial feature map using a convolutional neural network (CNN) for each frame of the sequence of the face gesture image, classifies the face feature map into the partial region, and performs LSTM. (Long ShortTerm Memory) The dynamic feature may be extracted between two objects based on a network, and the dynamic feature may be used to adaptively encode the importance of the relationship between the two objects.

The feature encoder may transfer the parameters of a plurality of convolution layers according to an image model learned with a large amount of face data and use the convolution neural network.

The object feature may be a fusion of the dynamic feature and a location feature of the corresponding object.

The relationship analyzer may analyze the facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc. by analyzing the relationship between the object features of the face gesture image based on the deep network. .

The following embodiments may provide a method and system for analyzing facial features using a facial motion relationship using the facial feature's relationship feature.

In detail, the following embodiments may analyze facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc. by analyzing the relationship between the dynamic features of the facial microregions based on the relationship analysis deep network. have.

Therefore, the following embodiments may be applicable to general face analysis techniques such as face authentication and facial expression recognition.

1 is a flowchart illustrating a facial characteristic analysis method using facial motion relations according to an exemplary embodiment of the present invention.
2 illustrates an example of a face analysis process using a relationship analysis deep network according to an exemplary embodiment of the present invention.
3 is a diagram for explaining the importance of relations regarding facial movements.
4 illustrates measurement results of measuring gender recognition accuracy according to an embodiment of the present invention.
5 is a block diagram illustrating a detailed configuration of a system for analyzing facial characteristics using facial motion relations according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

In addition, terms used in the present specification (terminology) are terms used to properly express preferred embodiments of the present invention, which may vary depending on the intention of the viewer, the operator, or customs in the field to which the present invention belongs. Therefore, the definitions of the terms should be made based on the contents throughout the specification.

The facial characteristic analysis method and system using the facial motion relation according to an embodiment of the present invention constitute a deep network using the facial motion relation, and encode a dynamic feature for each facial region to determine a difference between the relations of each region. It is about techniques to minimize. More specifically, the deep network proposed by the present invention encodes a dynamic feature by using a convolutional neural network (CNN) and a long short term memory (LSTM) for each face region, and calculates a relationship between dynamic features in each region. By analyzing the deep network and finally performing facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc., the main point is to improve the recognition accuracy and recognition rate compared to the conventional methods that did not consider the relationship.

Hereinafter, in the detailed description of the present invention, the present invention is limited to face dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc., but the present invention can be widely applied to general face analysis techniques such as facial authentication and facial expression recognition. It is apparent to those skilled in the art that there is.

Hereinafter, with reference to FIGS. 1 to 5, a method and a system for analyzing facial characteristics using a facial motion relationship according to an exemplary embodiment of the present invention will be described.

1 is a flowchart illustrating a facial characteristic analysis method using facial motion relations according to an exemplary embodiment of the present invention.

The operation method of FIG. 1 is performed by the facial characteristic analysis system using the face motion relationship according to the embodiment of the present invention shown in FIG. 5.

In operation 110, a face gesture image of the user is acquired. In this case, the face gesture image refers to a face image in which a user makes a specific gesture (expression).

In step 110, the facial characteristic analysis system using the facial motion relationship according to an embodiment of the present invention is a component that is provided with a camera device to directly obtain a face gesture image of each of the users, a plurality of users in conjunction with a plurality of terminals Receiving a face gesture image of each of the plurality of users from the terminal of each of the user, and receives the face gesture image of each of the plurality of users that is updated and stored in real time in the internal (or external) server (or database) It may include at least one or more of the components.

In operation 120, the dynamic feature of each region in the partial region of the face according to the facial movement is encoded from the facial gesture image.

For example, step 120 is a first step of extracting a facial feature map using a convolutional neural network (CNN) for each frame for a sequence of facial gesture images, and using the facial feature map as a partial region. Classifying and extracting a dynamic feature between two objects based on a Long Short Term Memory (LSTM) network; and a third step of adaptively encoding the importance of the relationship between the two objects by using the dynamic feature. It may include.

In this case, the first step is a facial feature using the image model trained with a large amount of face data as the convolutional neural network, for example, the transfer of parameters of a plurality of convolutional layers according to a VGGface model. You can extract the map. In the second step, the extracted facial feature map may be classified into a plurality of partial regions and used as an input of the LSTM network, and dynamic features of each of the plurality of partial regions may be extracted through the LSTM network. Here, the extracted dynamic feature may be used as an input of a relationship analysis deep network according to an embodiment of the present invention. In addition, the third step may adaptively encode the relational importance between the two objects in order to reflect the difference between the two objects in the partial region of the face according to the face movement.

In operation 130, the facial dynamic analysis may be analyzed by analyzing a relationship between object features including dynamic features of each region of the face using a deep network.

For example, in step 130, face dynamic analysis may be performed using Equation 1 below by using an object feature in which the extracted dynamic feature and the location feature of the corresponding object are fused.

[Equation 1]

및

는 각각 i번째 및 j번째 객체 특징을 나타내고, O는 모든 객체 특징의 집합을 의미한다. 또한,

및

는 각각 파라미터

와

의 함수로 다층 퍼셉트론(Multi­Layer Perceptron; MLP)에 의해 구현된 것일 수 있다. here,

And

Represents the i th and j th object features, respectively, and O represents the set of all object features. In addition,

And

Are each parameter

Wow

It may be implemented by a MultiLayer Perceptron (MLP) as a function of.

That is, in step 130, the facial characteristic analysis system using the facial motion relation according to an embodiment of the present invention analyzes the relationship between the object features of the facial gesture image based on the deep network, such as gender recognition, age estimation, At least one or more facial dynamic analysis of biometric authentication and facial expression recognition may be interpreted.

2 illustrates an example of a face analysis process using a relationship analysis deep network according to an exemplary embodiment of the present invention.

Referring to FIG. 2, in the facial characteristic analysis method 200 using the facial motion relationship according to an exemplary embodiment of the present invention, a convolutional neural network may be performed for each frame with respect to a smiling facial expression image, which is a facial gesture image 210 of a user. Facial feature maps can be extracted using Convolutional Neural Network (CNN). In this case, as a convolutional neural network, parameters of 10 convolutional layers of the VGG­face model trained with a large amount of face data may be transferred and used.

Subsequently, the facial characteristic analysis method 200 using the facial motion relation according to an embodiment of the present invention classifies the extracted facial feature map into nine facial regions and inputs them as inputs of a long short term memory (LSTM) network. In operation 220, the dynamic feature may be extracted and encoded.

Subsequently, in the facial feature analysis method 200 using the facial motion relation according to an embodiment of the present invention, the relationship analysis deeps an object feature 230 that fuses dynamic features extracted from nine face regions and location features of the corresponding object. The input to the network 240 may be used to interpret facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, and the like.

Accordingly, as shown in FIG. 2, in the facial characteristic analysis method 200 using the facial motion relation according to the embodiment of the present invention, the facial dynamic analysis result of gender recognition for the acquired facial gesture image 210 is performed. Derive (Gender: Female).

3 is a diagram for explaining the importance of relations regarding facial movements.

FIG. 3A illustrates a partial region of a face with respect to a face gesture image of the user, and FIG. 3B illustrates an object feature in the partial region.

As illustrated in FIG. 3, an object feature in a partial region of a face with respect to a face gesture image of a user may be extracted. In this case, the object feature may include a location feature defined as a center location for each face region and a dynamic feature of the location feature.

The facial characteristic analysis method and system using the facial motion relation according to an embodiment of the present invention can build a facial relational network based on a deep network for performing facial dynamic analysis using the object feature. . Face-relational networks provide a functional form of neural networks that extract key relationships for face-specific estimation, and consider the importance of relationships in face feature estimation.

As shown in FIG. 3, a relational feature representing a latent relation of any two objects extracted for face analysis is calculated by Equation 2 below.

[Formula 2]

은 학습 가능한 파라미터

를 갖는 함수를 나타내며,

는 i번째 내지 j번째 얼굴의 부분 영역에 대한 관계 쌍을 나타낸다. 또한, 두 객체 특징

의 관계에 대한 관계 중요성

는 하기의 [수식 3]과 같이 인코드(encode)된다.here,

Is a learnable parameter

Represents a function with

Denotes a relationship pair for the partial region of the i th to j th face. Also, two object features

Relationship importance for the relationship of

Is encoded as shown in Equation 3 below.

[Equation 3]

는 학습 가능한 파라미터

를 갖는 함수를 나타낸다.here,

Is a learnable parameter

Represents a function with

을 이용하여 두 개의 객체들 간의 관계 중요성

을 적응적으로 인코딩할 수 있으며, 이로 인해 얼굴 제스처 영상에 대한 얼굴 다이나믹 분석을 수행할 수 있다. 이 때, 얼굴 다이나믹 분석에 중요한 관계는 관계 중요성을 기반으로 획득될 수 있으며, 본 발명의 실시예에 따른 얼굴 움직임 관계를 이용한 얼굴 특성 분석 방법 및 그 시스템은 객체 특징과 얼굴 특성 추정의 관계 중요성을 결합할 수 있다. According to an embodiment of the present invention, there is provided a method and system for analyzing facial features using facial motion relations.

Importance of Relationship Between Two Objects Using

Can be adaptively encoded, which enables face dynamic analysis of facial gesture images. In this case, a relationship important for facial dynamic analysis may be obtained based on the relationship importance, and a method and system for analyzing facial features using facial motion relations according to an embodiment of the present invention may be based on the importance of the relationship between object characteristics and facial feature estimation. Can be combined.

For example, a high relationship importance value is important for evaluating facial features in partial regions of the face following facial movements. Basically, the importance of the relationship represents the importance of the relationship between two objects, so that the importance of the relationship between objects can be interpreted by combining the importance of the relationship between two objects.

4 illustrates measurement results of measuring gender recognition accuracy according to an embodiment of the present invention.

Referring to Figure 4, howold.net + dynamics method and COTS + dynamics method can be seen that shows the accuracy of 60.80% and 76.92% for people under 20 years of age, the proposed embodiment of the present invention It can be seen that the facial characteristic analysis method using the facial motion relation and the proposed relational network show 78.10% accuracy. Accordingly, it can be seen that the proposed present invention shows superior performance compared to the existing method.

Furthermore, the howold.net + dynamics method is 93.46% and the COTS + dynamics method is 93.00% for people over 20 years old, while the facial characteristic analysis method using the facial motion relationship according to the embodiment of the present invention and its The proposed relational network can be seen to recognize gender more accurately with 94.37% performance. In all three methods, the gender recognition accuracy for people under 20 years of age was lower than that for people over 20 years of age, indicating that for children, the differences between men and women are insignificant compared to adults men and women. You can think about it.

That is, the proposed method for analyzing facial features using a facial motion relationship and a proposed relational network according to an embodiment of the present invention propose a deep network structure for analyzing the relationship between movements in an effective fine face region, and comparing the experiments. The results verify that the performance is more accurate than the existing methods (howold.net + dynamics method, COTS + dynamics method) that did not consider the relationship.

5 is a block diagram illustrating a detailed configuration of a system for analyzing facial characteristics using facial motion relations according to an exemplary embodiment of the present invention.

Referring to FIG. 5, the facial characteristic analysis system using the facial motion relation according to the embodiment of the present invention performs face analysis based on the relation between the movements in each facial region.

To this end, the facial characteristic analysis system 500 using the facial motion relationship according to the embodiment of the present invention includes an image acquisition unit 510, a feature encoding unit 520, and a relationship analysis unit 530.

The image acquisition unit 510 acquires a face gesture image of the user. In this case, the face gesture image refers to a face image in which a user makes a specific gesture (expression).

The facial characteristic analysis system 500 using the facial motion relationship according to an embodiment of the present invention includes a component that is provided with a camera device to directly obtain a facial gesture image of each of the users, and is linked with a plurality of terminals of each of the users. Receiving a face gesture image of each of the plurality of users from the terminal of each of the user, and receives the face gesture image of each of the plurality of users that is updated and stored in real time in the internal (or external) server (or database) It may include at least one or more of the components.

The feature encoder 520 encodes dynamic features for respective regions in the partial region of the face according to the face movement from the face gesture image.

For example, the feature encoder 520 extracts a facial feature map using a convolutional neural network (CNN) for each frame of a sequence of facial gesture images, and extracts a facial feature map from a partial region. By classifying as, it is possible to extract dynamic features between two objects based on long short term memory (LSTM) network, and to adaptively encode the importance of the relationship between two objects.

At this time, the feature encoding unit 520 is a convolutional neural network using an image model trained with a large amount of face data, for example, using a transition train of parameters of a plurality of convolution layers according to a VGGface model. The facial feature map can be extracted. In addition, the feature encoding unit 520 classifies the extracted facial feature map into a plurality of partial regions and uses them as inputs to the LSTM network, and extracts dynamic features for each of the plurality of partial regions through the LSTM network. have. Here, the extracted dynamic feature may be used as an input of a relationship analysis deep network according to an embodiment of the present invention.

The relationship analyzer 530 analyzes the relationship between the object features including the dynamic features of each region of the face by using a deep network to analyze the face dynamic analysis.

For example, the relationship analyzer 530 may perform facial dynamic analysis through [Equation 1] using an object feature in which the extracted dynamic feature and the location feature of the corresponding object are fused. Accordingly, the relationship analyzer 530 analyzes a relationship between object features of a face gesture image based on a deep network and analyzes at least one or more face dynamics among gender recognition, age estimation, biometric authentication, and facial expression recognition. Can be interpreted.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CDROMs, DVDs, and magneto-optical media such as floppy disks. (magnetooptical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or other components. Or even if replaced or replaced by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

200, 500: Relationship analysis method and system
210: facial gesture image
230: object features

Claims (12)

Obtaining a face gesture image of a user;
Encoding dynamic features for respective regions in the partial region of the face according to facial movement from the face gesture image; And
Analyzing a facial dynamic analysis by analyzing a relationship between object features including the dynamic features of each region of the face by using a deep network,
Encoding the dynamic feature for each region is
Extracting a facial feature map using a convolutional neural network (CNN) for each frame of the face gesture image;
Classifying the facial feature map into nine sub-regions and extracting the dynamic feature between two objects based on a long short-term memory (LSTM) network; And
Adaptively encoding the importance of the relationship between the two objects using the dynamic feature,
Adaptively encoding the importance of the relationship between the two objects
Adaptively encode the relational importance between the two objects in order to reflect the difference between the two objects in the partial region of the face according to the movement of the face,
Interpreting the facial dynamic analysis
Analyzing the facial dynamic analysis of gender recognition, age estimation, biometric authentication and facial expression recognition by analyzing the relationship between the object features of facial movements on the face gesture image based on the deep network. , Facial feature analysis using face motion relationship. delete The method of claim 1,
Extracting the facial feature map
A facial characteristic analysis method using a facial motion relationship, characterized in that the transfer of a plurality of parameters (convolution) of the convolutional layer according to the image model learned with a large amount of face data to use as the convolutional neural network. The method of claim 1,
Extracting the dynamic feature
The extracted facial feature map is classified into a plurality of partial regions of the face and used as an input of the LSTM network, and the dynamic feature is extracted from the LSTM network. Analytical Method. The method of claim 1,
The object feature is
Facial characteristic analysis method using the facial motion relationship characterized in that the fusion of the dynamic feature and the location feature of the object. delete delete An image obtaining unit obtaining a face gesture image of the user;
A feature encoding unit for encoding a dynamic feature for each region in a partial region of a face according to face movement from the face gesture image; And
It includes a relationship analysis unit for analyzing the relationship between the object features including the dynamic feature of each region of the face using a deep network to analyze the face dynamic analysis,
The feature encoding unit
For each frame of the face gesture image, a facial feature map is extracted using a convolutional neural network (CNN) for each frame, and the face feature map is classified into nine sub-regions to form an LSTM (Long). Short-Term Memory) extracts the dynamic feature between two objects based on a network, and uses the dynamic feature to reflect the difference between the two objects in the partial region of the face according to the movement of the face. Adaptively encodes relational importance,
The relationship analysis unit
Analyzing the facial dynamic analysis such as gender recognition, age estimation, biometric authentication, facial expression recognition, etc. by analyzing the relationship between the object features of the facial motion on the face gesture image based on the deep network. A facial characteristic analysis system using a facial motion relationship. delete The method of claim 8,
The feature encoding unit
A facial characteristic analysis system using a facial motion relationship, characterized in that the transfer of the parameters of the plurality of convolution layers according to the image model learned with a large amount of face data to use as the convolutional neural network. The method of claim 8,
The object feature is
Facial characteristic analysis system using the facial motion relationship characterized in that the fusion of the dynamic feature and the location feature of the object. delete

Images