JP2022046419A - Image processing apparatus, image processing method, and machine readable storage medium - Google Patents

Abstract

To provide an image processing apparatus, an image processing method, and a machine readable storage medium in the present invention.SOLUTION: An image processing apparatus includes: an information acquisition section that divides an image to be input into a plurality of areas and acquires information on face action units in the plurality of areas; an action unit characteristic extraction section that extracts action unit characteristics from the areas of the face action units based on the acquired information on the face action units; a face global characteristic extraction section that extracts face global characteristics based on the acquired information on the face action units; and a classification section that classifies the face action units based on the action unit characteristics and the face global characteristics.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technical field of image processing, and more particularly to an image processing apparatus for recognizing facial expression, an image processing method, and a machine-readable storage medium.

The face always has a great deal of information about a person's mental and emotional state. Therefore, the mental state can be quantified based on the recognition of facial expression. Recognition of facial expression has many uses in real life. For example, paying attention to the psychology of employees and improving their motivation can improve labor productivity, and estimating customer satisfaction can improve their purchasing motivation (digital). Marketing) or the driving situation of the driver can be estimated.

Therefore, it is very important to develop a device that can effectively recognize facial expression.

An object of the present invention is an image processing device and an image processing method capable of recognizing a facial expression based on detection of at least the appearance of an action unit (Action Unit (AU)). And to provide machine-readable storage media.

According to one aspect of the invention, an image processing apparatus is provided, which is:
An information acquisition unit that divides the input image into multiple areas and acquires information about face action units in multiple areas;
Action unit feature extraction unit that extracts action unit features from the area of the face action unit based on the acquired information about the face action unit;
A face global feature extraction unit that extracts face global features based on the acquired information about the face action unit; and a classification unit that classifies face action units based on both the action unit features and the face global features. include.

According to another aspect of the invention, an image processing method is provided, which is:
Divide the input image into multiple areas and get information about face action units in multiple areas;
Based on the acquired information about the face action unit, the action unit features are extracted from the area of the face action unit;
It involves extracting face global features based on the information about the acquired face action units; and classifying the face action units based on both the action unit features and the face global features.

According to another aspect of the present invention, a machine-readable storage medium is provided, in which a program product containing a machine-readable command code is carried, and the command code is read by a computer. When executed, the computer can be made to execute the image processing method according to the present invention.

According to the image processing apparatus, the image processing method, and the machine-readable storage medium according to the present invention, a microexpression can be recognized by detecting the appearance of a facial action unit corresponding to each local region of the face portion.

It is a block diagram of the structural block of the image processing apparatus in the Example of this invention. It is a block diagram of the structure of the information acquisition part in the image processing apparatus in the Example of this invention. It is a block diagram of the structure of the action unit feature extraction part in the image processing apparatus in the Example of this invention. It is a block diagram of the face global feature extraction part in the image processing apparatus in the Example of this invention. It is a block diagram of the structural block of the image processing apparatus according to another embodiment of this invention. It is a figure which shows that the action unit area determination part in the image processing apparatus by another embodiment of this invention determines a new area. It is a flowchart of the image processing method in an Example of this invention. It is an exemplary block diagram of a general-purpose personal computer that can realize the image processing apparatus and method in the embodiment of the present invention.

Hereinafter, suitable examples for carrying out the present invention will be described in detail with reference to the accompanying drawings. It should be noted that such examples are merely examples and do not limit the present invention.

Hereinafter, how the image processing apparatus according to the embodiment of the present invention recognizes facial expression is described with reference to FIG. 1.

FIG. 1 is a block diagram of a configuration of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 100 in the embodiment of the present invention may include an information acquisition unit 110, an action unit feature extraction unit 120, a face global feature extraction unit 130, and a classification unit 140. Here, it is assumed that the input image has only one face. The facial expression recognition task is to detect the appearance of all facial action units of user interest. Whether or not a facial action unit appears can indicate whether or not there is movement of facial muscles in the facial region corresponding to the facial action unit.

First, the information acquisition unit 110 can divide the input image into a plurality of areas and acquire information about the face action unit in the plurality of areas. Further, the information acquisition unit 110 can provide the action unit feature extraction unit 120 and the face global feature extraction unit 130 with information about the face action unit in the acquired plurality of areas.

Further, the action unit feature extraction unit 120 can extract the action unit feature from the area of the face action unit based on the information about the face action unit provided by the information acquisition unit 110. Further, the action unit feature extraction unit 120 can provide the extracted action unit features to the classification unit 140.

Further, the face global feature extraction unit 130 can extract the face global feature based on the information about the face action unit provided by the information acquisition unit 110. Further, the face global feature extraction unit 130 can provide the extracted face global feature to the classification unit 140.

Further, the classification unit 140 classifies the face action unit based on both the action unit feature provided by the action unit feature extraction unit 120 and the face global feature provided by the face global feature extraction unit 130. Can be done.

Therefore, according to the image processing apparatus 100 in the embodiment of the present invention, it is possible to provide an effective end-to-end apparatus, which detects the appearance of a facial action unit corresponding to each local region of the face. It is possible to recognize facial expression. Further, since the image processing device 100 can utilize both the action unit feature and the face global feature, better detection results can be obtained.

Hereinafter, the configurations of the information acquisition unit 110, the action unit feature extraction unit 120, and the face global feature extraction unit 130 in the image processing device 100 according to the embodiment of the present invention will be described with reference to FIGS. 2 to 4.

FIG. 2 is a block diagram of the information acquisition unit 110 in the image processing device 100 according to the embodiment of the present invention. In FIG. 2, the information acquisition unit 110 is an information acquisition unit 200 as an example.

The information acquisition unit 200 has a plurality of convolution layers, and is used to obtain information about a face action unit in a plurality of regions in an input human face image. Preferably, the information about the face action unit may include information indicating the position of the face action unit in the image. Preferably, the information acquisition unit 200 can acquire information about the face action unit by converting the image into a feature map in a low-dimensional space.

As shown in FIG. 2, the input image is provided to the two-dimensional convolution block 210. The Conv block 210 can perform feature learning on the input image and perform abstraction by performing a convolution operation on the input image. Further, the image processed by the Conv block 210 is input to the division block 220, and the division block 220 can divide the input image into n areas.

Then, each of the n regions is a set of batch normalization (BatchNorm, batch normalization) blocks 230-1 to 230-n, and parametric ReLU (PreLU, parametric rectified linear unit) blocks 240-1 to 240-n. , And the Conv blocks 250-1 to 250-n for processing. Here, i in BatchNorm block 230-i, PRELU block 240-i, and Conv block 250-i shown in FIG. 2 is an integer larger than 1 and smaller than n.

The processing results output by the n sets of BatchNorm blocks, PRELU blocks, and Conv blocks are provided to the joining block 260, and the joining block 260 can join the processing results of the n regions. For example, the stitching block 260 can stitch the processing results of the n regions corresponding to the division of the n regions by the split block 220. The result processed by the superposition block 260 is then provided to the superposition block 270. The superimposition block 270 superimposes the image processed by the Conv block 210 and the result processed by the splicing block 260, and superimposes the superimposed results on the BatchNorm block 280, the PRELU block 290 and the maximum pooling. (Maxpooling) Block 295 is sequentially provided for further processing.

Finally, the information acquisition unit 200 provides the feature map as an output to the action unit feature extraction unit 120 and the face global feature extraction unit 130 shown in FIG.

For example, the information acquisition unit 200 may be a uniform block learning module as an example, and n is 64 because the uniform block learning module can divide an input image into, for example, 8 × 8 regions. Each of the 64 regions passes through a set of BatchNorm blocks, PRELU blocks and Conv blocks for processing. After that, the output feature maps of each set of BatchNorm blocks, PRELU blocks, and Conv blocks are stitched together, and after the superposition operation as described above is performed, the BatchNorm blocks 280, PRELU blocks 290, and Maxpooling blocks 295 are sequentially supplied. Will be done. Here, the final output feature map is also divided into 8 × 8 units. In such a case, the output feature map is also referred to as a uniform feature map.

FIG. 3 is a block diagram of the action unit feature extraction unit 120 in the image processing device 100 according to the embodiment of the present invention. In FIG. 3, the action unit feature extraction unit 120 is an action unit feature extraction unit 300 as an example.

The action unit feature extraction unit 300 can extract action unit features from the area of the face action unit based on the information about the face action unit provided by the information acquisition unit. The action unit feature extraction unit 300 may be composed of several convolution layers, and these convolution layers participate only in the action unit region. Preferably, the action unit feature extraction unit 300 can independently extract the action unit features for the region of each face action unit by the plurality of convolution layers. The action unit feature may be, for example, the final feature map of the AU region of the human face.

For example, as shown in FIG. 3, the action unit feature extraction unit 300 is provided with a feature map of m AU regions of a human face image. More specifically, each of the m feature maps of the AU regions is a set of Conv blocks 310-1 to 310-m, BatchNorm blocks 320-1 to 320-m, and PRELU blocks 330-1 to 330-m. And the results after being processed by Conv blocks 310-1 to 310-m, BatchNorm blocks 320-1 to 320-m and PRELU blocks 330-1 to 330-m are the corresponding Maxpolling blocks 370-1 respectively. It is input to ~ 370-m. Then, after the processing of Maxpooling blocks 370-1 to 370-m, the final feature map of m AU regions on the human face can be obtained. Preferably, m is 12, i.e., the input is the initial feature map of the 12 AU regions of the human face, and the output is the final feature map of the 12 AU regions of the human face. ..

For each AU area, it is possible to process the input feature map of the AU area by using a plurality of sets of Conv blocks, BatchNorm blocks, and PRELU blocks. In this way, the action unit features can be processed. Further optimization can help improve the accuracy of the subsequent classification process of the classification unit 140. For example, in FIG. 3, one set of Conv blocks 340-1 to 340-m, BatchNorm blocks 350-1 to 350-m, and PRELU blocks 360-1 to 360-m are increased.

FIG. 4 is a block diagram of the face global feature extraction unit 130 in the image processing device 100 according to the embodiment of the present invention. In FIG. 4, the face global feature extraction unit 130 is, for example, the face global feature extraction unit 400.

The face global feature extraction unit 400 in FIG. 4 can extract face global features based on the information about the face action unit provided by the information acquisition unit. Face global feature learning consists of several convolutional layers, which spread over the entire face. Preferably, the face global feature extraction unit 400 can extract the face global feature by the plurality of convolution layers based on the feature map provided by the information acquisition unit.

As shown in FIG. 4, the face global feature extraction unit 400 provides a feature map of a human face image, for example, a uniform feature map. Specifically, the feature map is input to the Conv block 410, the BatchNorm block 420 and the PRELU block 430, and the result after the processing of the Conv block 410, the BatchNorm block 420 and the PRELU block 430 is input to the Maxpolling block 470. Then, after being processed by the Maxpooling block 470, the global feature map of the human face can be acquired.

It should be noted that the input feature map can be processed by using a plurality of sets of Conv block, BatchNorm block and PRELU block, and in this way, further optimization is performed for the face global feature. , It is possible to help improve the accuracy of the subsequent classification process of the classification unit 140. For example, in FIG. 4, a set of Conv blocks 440, BatchNorm blocks 450 and PRELU blocks 460 are increasing.

Subsequently, the operation of the classification unit 140 in FIG. 1 will be described. The classification unit 140 can detect the appearance of an action unit based on the combination of the action unit feature and the face global feature. For example, the classification unit 140 may be a fully connected neural network. Alternatively, the classifier 140 may consist of very few linear layers with a softmax output, which output is the probability of appearance of all action units. In other words, the classification unit 140 can classify the face action units by analyzing the probability of appearance of all the face action units. Preferably, the classification unit 140 can analyze the probability of appearance of all facial action units using a linear layer with a softmax output.

The use of the image processing apparatus 100 in the embodiment of the present invention has two stages, that is, a training stage and an evaluation stage. In the training phase, the network is trained using the face image labeled with the appearance of the action unit, and in the evaluation phase, the trained network is used to detect the movement in the test facial image and evaluate the performance. For example, in the training stage, facial images, feature information related to facial feature parts (for example, eyebrows, corners of the mouth, etc.), linkage annotations related to the face action unit, etc. are input to the image processing device shown in FIG. 1 or shown in FIG. 5, which will be described later. After being processed by the image processing device, the prediction probabilities of different face action units can be obtained.

Hereinafter, the image processing apparatus 500 according to another embodiment of the present invention will be described with reference to FIGS. 5 and 6.

FIG. 5 is a block diagram of the image processing apparatus 500 according to another embodiment of the present invention, and FIG. 6 is an action unit area determination unit 550 in the image processing apparatus 500 according to another embodiment of the present invention. Is a diagram showing that a new region is determined.

As shown in FIG. 5, the image processing apparatus 500 in another embodiment of the present invention may further include an action unit region determination unit 550. The information acquisition unit 510, the action unit feature extraction unit 520, the face global feature extraction unit 530, and the classification unit 540 shown in FIG. 5 are the information acquisition unit 110, the action unit feature extraction unit 120, and the face global feature extraction unit 130, respectively, as shown in FIG. And for the classification unit 140. Therefore, the description of the information acquisition unit, the action unit feature extraction unit, the face global feature extraction unit, and the classification unit based on FIGS. 1 to 4 also includes the information acquisition unit 510 and the action unit feature extraction unit 520 in the present embodiment. It can be applied to the description of the face global feature extraction unit 530 and the classification unit 540. Therefore, the detailed description thereof will be omitted here.

Specifically, the information acquisition unit 510 can divide the input image into a plurality of areas and acquire information about the face action unit in the plurality of areas. In addition, the information acquisition unit 510 can provide information about the face action unit in the acquired plurality of areas to the action unit area determination unit 550 and the face global feature extraction unit 530.

Further, the action unit area determination unit 550 can redetermine a new area of the face action unit based on the information (for example, feature map) about the face action unit provided by the information acquisition unit 510. Further, the action unit area determination unit 550 can provide the determined new area and information regarding the new area to the action unit feature extraction unit 520.

Further, the action unit feature extraction unit 520 can extract the action unit features from the new area of the face action unit based on the new area of the face action unit provided by the action unit area determination unit 550 and the information about the new area. .. Further, the action unit feature extraction unit 520 can provide the extracted action unit features to the classification unit 540.

Further, the face global feature extraction unit 530 can extract the face global feature based on the information about the face action unit provided by the information acquisition unit 510. Further, the face global feature extraction unit 530 can provide the extracted face global feature to the classification unit 540.

Further, the classification unit 540 classifies the face action unit based on both the action unit feature provided by the action unit feature extraction unit 520 and the face global feature provided by the face global feature extraction unit 530. Can be done.

Therefore, the difference from the embodiment shown in FIG. 1 is that the information acquisition unit 510 provides the acquired information on the face action unit in the plurality of regions to the action unit region determination unit 550 and the face global feature extraction unit 530, and The action unit feature extraction unit 520 extracts the action unit feature from the new area determined by the action unit area determination unit 550.

Preferably, the action unit area determination unit 550 determines a new area for each face action unit before the action unit feature extraction unit 520 extracts the action unit features, the new area including only the face area in the image, and , The action unit feature extraction unit 520 extracts the action unit features based on the determined new area.

Preferably, the action unit area determination unit 550 determines a new area for each face action unit by the following method, that is, constructs a three-dimensional (3D) face model based on the face action unit in the image, and then By modifying the area of the face action unit using the 3D face model, the area of each modified face action unit is made to include only the face area in the image.

For example, the action unit area determination unit 550 first constructs a 3D face model from the original image using 3DMM, and then determines a new area for each action unit based on the 3D face model. This means constructing a specific 3D face model by using posture and marking as parameters of the existing 3DMM. The 3DMM model is unique and needs to provide orientation and markings to build different 3D models for a particular image.

As shown in FIG. 6, the figure on the left is the area of the face action unit before modification, which includes parts of the image that do not belong to the face area, such as hair, background, etc., and the figure on the right is the area of face action after modification. The area of the unit, which contains only the face area in the image. FIG. 6 shows that the action unit area determination unit 550 modified the area of the face action unit related to the area of the outer corner of the eye. Alternatively, the action unit area determination unit 550 can modify the area of the face action unit related to other areas of the face.

In the actual shooting of the image, the angle of the camera is different, so in the shot human face image, the face may move to a specific side instead of facing forward. In such a case, the image processing device 500 having the action unit area determination unit 550 is used to correct the area of the face action unit to eliminate the influence of the shooting angle and to improve the characteristics of the action unit. An effective estimate can be provided.

Therefore, according to the image processing device 500 in the embodiment of the present invention, an effective end-to-end device capable of automatically performing facial expression recognition that is robust to changes in posture (face orientation) can be provided. The image processing device can recognize a facial expression by detecting the appearance of a facial action unit corresponding to each local region of the face portion, regardless of the posture of the head.

Hereinafter, the image processing method in the embodiment of the present invention will be described with reference to FIG. 7.

As shown in FIG. 7, the image processing method in the embodiment of the present invention starts in step S110. In step S110, the input image is divided into a plurality of areas, and information about the face action unit in the plurality of areas is acquired.

Subsequently, in step S120, the action unit feature is extracted from the area of the face action unit based on the acquired information about the face action unit.

Subsequently, in step S130, the face global feature is extracted based on the acquired information about the face action unit.

Subsequently, in step S140, the face action unit is classified based on both the action unit feature and the face global feature. After that, the process ends.

Therefore, according to the image processing method in the embodiment of the present invention, facial expression recognition can be performed by detecting the appearance of a facial action unit corresponding to each local region of the face portion.

In an embodiment of the invention, the method further comprises determining a new region for each facial action unit prior to extracting the action unit features, the new region comprising only the facial region in the image and determining. Extract action unit features based on the new area created.

In an embodiment of the invention, the step of determining a new region for each face action unit builds a three-dimensional (3D) face model based on the face action unit in the image, and uses the 3D face model to build the face action unit. This includes modifying the area of the face so that the area of each modified face action unit includes only the face area in the image.

Therefore, according to the image processing method in the embodiment of the present invention, it is possible to automatically perform facial expression recognition that is robust to changes in posture (face orientation) regardless of the posture of the head.

In the embodiment of the present invention, the information about the face action unit includes information indicating the position of the face action unit in the image.

In an embodiment of the invention, acquisition of information about a face action unit involves transforming an image into a feature map in low dimensional space.

In the embodiment of the present invention, the extraction of the action unit feature includes extracting the action unit feature independently for each face action unit region by a plurality of convolution layers.

In the embodiment of the present invention, the extraction of the face global feature includes extracting the face global feature by a plurality of convolution layers based on the feature map.

In the embodiments of the present invention, the plurality of convolution layers includes at least a batch normalization unit, a parametric PRELU, a two-dimensional convolution unit and a maximum pooling unit.

In the embodiments of the present invention, the classification for facial action units involves analyzing the probability of appearance of all facial action units.

In an embodiment of the invention, a linear layer with a softmax output is used to analyze the probability of appearance of all facial action units.

Since various specific implementation methods of the above-mentioned steps of the image processing method in the embodiment of the present invention have been described in detail above, the detailed description thereof will be omitted here.

As is clear, each operational process of the image processing method of the present invention can be realized by a computer-executable program stored in various machine-readable storage media.

Further, an object of the present invention may be realized by the following method, that is, a storage medium in which the above-mentioned executable program code is stored is directly or indirectly provided to a system or an apparatus, and the system is provided. Alternatively, the computer or central processing unit (CPU) in the device reads and executes the above-mentioned program code. At this time, if the system or device has a function capable of executing a program, the implementation method of the present invention is not limited to the program, and the program is executed by any form, for example, an object-oriented program or an interpreter. It may be a program to be executed, a script program provided to the operating system, or the like.

These machine-readable storage media may include, but are limited to, various memories and storage units, semiconductor devices, magnetic disks such as magneto-optical, magneto-optical disks, and other media suitable for storing information. do not have.

Further, the computer connects to a corresponding website on the Internet, downloads and installs the computer program code according to the present invention on the computer, and then executes the program to realize the technical proposal of the present invention. You can also do it.

FIG. 8 is a structural diagram of a hardware configuration (general-purpose machine) 1300 that can realize the information processing method and device according to the embodiment of the present invention.

The general-purpose machine 1300 may be, for example, a computer system. The general-purpose machine 1300 is merely an example, and does not limit the applicable range or function of the method and device according to the present invention. Further, the general-purpose machine 1300 does not depend on any module, assembly, or a combination thereof in the above-mentioned method and device.

In FIG. 13, the central processing unit (CPU) 1301 performs various processes based on the program stored in the ROM 1302 or the program rodged from the storage unit 1308 to the RAM 1303. The RAM 1303 can also store data and the like necessary for the CPU 1301 to perform various processes according to needs. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via the bus 1304. The input / output interface 1305 is also connected to bus 1304.

Further, the following components are further connected to the input / output interface 1305, that is, an input unit 1306 including a keyboard and the like, a display unit such as a liquid crystal display (LCD), and an output unit including a speaker and the like. 1307, a storage unit 1308 including a hard disk and the like, and a communication unit 1309 including a network interface card such as a LAN card and a modem. The communication unit 1309 performs communication processing via a network such as the Internet or LAN. Drive 1310 may be connected to the input / output interface 1305, if desired. The removable medium 1311 such as a semiconductor memory can be set in the drive 1310 as needed, so that the computer program read from the medium can be installed in the storage unit 1308.

The present invention also provides a program product including a machine readable command code. When such a command code is read and executed by a machine, the method according to the embodiment of the present invention described above can be executed. Correspondingly, such program products are carried, for example, magnetic disks (including floppy disks (registered trademarks)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (MD (registered trademarks)). ), And various storage media such as semiconductor storage devices are also included in the present invention.

The above-mentioned storage medium may include, but is not limited to, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor storage device, and the like.

Further, each operation (process) in the above method can be realized by a method of a computer-executable program stored in various machine-readable storage media.

In addition, the above examples will be further disclosed as an appendix as follows.

(Appendix 1)
It is an image processing device
An information acquisition unit that divides an input image into a plurality of areas and obtains information about a face action unit in the plurality of areas;
An action unit feature extraction unit that extracts action unit features from the area of the face action unit based on the acquired information about the face action unit;
The face global feature extraction unit that extracts the face global feature based on the acquired information about the face action unit; and the classification for the face action unit based on both the action unit feature and the face global feature. A device that includes a classifier to perform.

(Appendix 2)
The image processing apparatus according to Appendix 1, further.
The action unit feature extraction unit includes an action unit area determination unit that determines a new area for each face action unit before extracting the action unit feature.
The new area includes only the face area in the image.
The action unit feature extraction unit is a device that extracts the action unit features based on a determined new area.

(Appendix 3)
The image processing apparatus according to Appendix 2,
The action unit area determination unit determines a new area for each face action unit by the following method, that is, a three-dimensional (3D) face model is constructed based on the face action unit in the image, and the face model is described. A device that modifies the area of the face action unit using a 3D face model so that the area of each modified face action unit includes only the face area in the image.

(Appendix 4)
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is described.
The device including information indicating the position of the face action unit in the image of the face action unit.

(Appendix 5)
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is described.
The information acquisition unit is a device that obtains information about the face action unit by converting the image into a feature map in a low-dimensional space.

(Appendix 6)
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is described.
The action unit feature extraction unit is a device that independently extracts the action unit features for each face action unit area by a plurality of convolution layers.

(Appendix 7)
The image processing apparatus according to Appendix 5, wherein the image processing apparatus is described.
The face global feature extraction unit is a device that extracts the face global feature based on the feature map by a plurality of convolution layers.

(Appendix 8)
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is described.
The classification unit is a device that classifies the face action units by analyzing the probability of appearance of all the face action units.

(Appendix 9)
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is described.
The classification unit analyzes the probability of appearance of all the facial action units using a linear layer having a softmax output (Appendix 10).
It ’s an image processing method.
The input image is divided into a plurality of areas, and information about the face action unit in the multiple areas is acquired;
Based on the acquired information about the face action unit, the action unit features are extracted from the area of the face action unit;
Extracting face global features based on the acquired information about the face action unit; and including classifying the face action units based on both the action unit features and the face global features. Method.

(Appendix 11)
The method described in Appendix 10
Before extracting the action unit features,
Including establishing a new area for each face action unit
The new area includes only the face area in the image.
A method of extracting said action unit features based on a defined new area.

(Appendix 12)
The method described in Appendix 11
Determining a new area for each face action unit is
A three-dimensional (3D) face model is constructed based on the face action unit in the image, and the area of the face action unit is modified by using the 3D face model, and the area of each modified face action unit is described as described above. A method to include only the face area in the image.

(Appendix 13)
The method described in Appendix 10
The method, wherein the information about the face action unit includes information indicating the position of the face action unit in the image.

(Appendix 14)
The method described in Appendix 10
Acquiring information about the face action unit comprises converting the image into a feature map in a low dimensional space.

(Appendix 15)
The method described in Appendix 10
The method comprising extracting the action unit features independently for each face action unit region by means of a plurality of convolution layers.

(Appendix 16)
The method described in Appendix 14,
A method comprising extracting the face global feature from a plurality of convolution layers based on the feature map.

(Appendix 17)
The method according to Appendix 15 or 16.
The method, wherein the multi-convolution layer comprises at least a batch normalization unit, a parametric PRELU, a two-dimensional convolution unit and a maximum pooling unit.

(Appendix 113)
The method described in Appendix 10
The classification for the facial action unit comprises analyzing the probability of appearance of all facial action units.

(Appendix 19)
The method described in Appendix 18
A method of analyzing the probability of appearance of all of the facial action units by a linear layer with a softmax output.

(Appendix 20)
It is a machine-readable storage medium
A program product including a machine-readable command code is stored in the program product, and when the command code is read and executed by a computer, the image processing method described in Appendix 10-19 is executed on the computer. A machine-readable storage medium that can be made to.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention belongs to the technical scope of the present invention as long as the gist of the present invention is not deviated.

Claims (10)

It is an image processing device
An information acquisition unit that divides an input image into a plurality of areas and acquires information about a face action unit in the plurality of areas;
An action unit feature extraction unit that extracts action unit features from the area of the face action unit based on the acquired information about the face action unit;
A face global feature extraction unit that extracts face global features based on the acquired information about the face action unit; and a classification for the face action unit based on both the action unit features and the face global features. A device that includes a classifier to perform. The image processing apparatus according to claim 1.
The action unit feature extraction unit further includes an action unit area determination unit that determines a new area for each face action unit before the action unit feature extraction unit extracts the action unit feature.
The new area includes only the face area in the image.
The action unit feature extraction unit is a device that extracts the action unit features based on a determined new area. The image processing apparatus according to claim 2.
The action unit area determination unit constructs a three-dimensional (3D) face model based on the face action unit in the image, and modifies the area of the face action unit using the 3D face model, and after the modification. A device that determines a new area for each face action unit by allowing the area of each face action unit in the image to include only the face area in the image. The image processing apparatus according to claim 1.
The device including information indicating the position of the face action unit in the image of the face action unit. The image processing apparatus according to claim 1.
The information acquisition unit is a device that obtains information about the face action unit by converting the image into a feature map in a low-dimensional space. The image processing apparatus according to claim 1.
The action unit feature extraction unit is a device that independently extracts the action unit features for each face action unit area by a plurality of convolution layers. The image processing apparatus according to claim 5.
The face global feature extraction unit is a device that extracts the face global feature based on the feature map by a plurality of convolution layers. The image processing apparatus according to claim 1.
The classification unit is a device that classifies the face action units by analyzing the probability of appearance of all the face action units. It ’s an image processing method.
The input image is divided into a plurality of areas, and information about the face action unit in the multiple areas is acquired;
Based on the acquired information about the face action unit, the action unit features are extracted from the area of the face action unit;
Extracting face global features based on the acquired information about the face action unit; and including classifying the face action units based on both the action unit features and the face global features. Method. A program for causing a computer to execute the image processing method according to claim 9.

Images