JP6788264B2 - Facial expression recognition method, facial expression recognition device, computer program and advertisement management system - Google Patents

Description

The present invention relates to a facial expression recognition method, a facial expression recognition device, a computer program, and an advertisement management system. Specifically, the present invention relates to an image processing technique for improving the accuracy of facial expression recognition using a hierarchical convolutional neural network.

In recent years, the performance of image recognition by deep learning has been dramatically improved. Deep learning is a general term for machine learning using a multi-layered hierarchical neural network. As the multi-layered hierarchical neural network, for example, a convolutional neural network (hereinafter, also referred to as “CNN”) is used.
CNN has a multi-layered laminated structure in which a convolutional layer and a pooling layer in a local region are repeated, and it is said that such a laminated structure improves image recognition performance.

As shown in Non-Patent Document 1, deep learning using a convolutional neural network has already been performed to recognize universal facial expression classes such as happiness, surprise, fear, sadness, anger, and disgust. There is.

"Acquisition of facial expression expression using convolutional neural network" Daiki Nishimei and 4 others 2016 Annual Meeting of the Japanese Society for Artificial Intelligence 4L1-5in1 General presentation on June 9, 2016

In facial expression recognition using a convolutional neural network, the pixel value (RGB value) of the original image is input to the network as it is without preprocessing the original image of the face, or the pixel value is subjected to principal component analysis (Principle Component Analysis). Is done.
For example, in Non-Patent Document 1, GCN (Global Contrast Normalization) is executed as preprocessing for the original image of the face.

As described above, conventionally, the pixel value (raw data) of the original image is used as it is, or only preprocessing for extracting a single feature factor from the original image is performed. This point is considered to be one of the causes of suppressing the improvement of the accuracy of facial expression recognition.
In view of the conventional problems, an object of the present invention is to improve the accuracy of facial expression recognition using a hierarchical neural network.

(1) The facial expression recognition method of the present invention is a method for recognizing facial facial expressions included in a captured image, and is a learning image group having at least three types of information including facial unevenness information, texture information, and contour information. As input data, a learning step of causing a hierarchical neural network to learn parameters, and extracting features related to at least the three types of information from the captured image to generate a plurality of input images, and generating the plurality of input images. Is included as input data, and includes a recognition step of causing the trained hierarchical neural network to recognize the facial expression included in the captured image.

According to the facial expression recognition method of the present invention, in the learning step, the input data used for learning the parameters of the hierarchical neural network has features related to at least three types of information including facial unevenness information, texture information and contour information. It consists of a group of images for learning.
Further, in the recognition step, the input data for facial expression recognition by the trained hierarchical neural network is composed of a plurality of input images generated by extracting features related to at least the above three types of information from the captured image.

Therefore, the accuracy of facial expression recognition using the hierarchical neural network is higher than that of the conventional technique in which no preprocessing is performed before input to the hierarchical neural network or preprocessing for extracting only a single feature factor is performed. Can be improved (see FIG. 8).

(2) In the facial expression recognition method of the present invention, specifically, the hierarchical neural network comprises a convolutional neural network.
The reason is that the convolutional neural network can realize high performance for image recognition including facial expression recognition.

(3) In the expression recognition method of the present invention, the unevenness information is the direction angle of the gradient vector of the pixel value at each pixel point, and the texture information is the norm of the gradient vector of the pixel value at each pixel point. The contour information is preferably position information of pixel points whose pixel values change sharply.
The reason is that the above-mentioned direction angle (A), gradient vector norm (G) and contour information (E) can be calculated by existing open source software. Therefore, if these parameters are adopted, the present invention Is relatively easy to implement.

(4) In the expression recognition method of the present invention, the learning step specifically generates the learning image group by extracting features related to at least the three types of information from a plurality of sample images. The step includes an update step of updating the parameter of the network based on the recognition result output by the hierarchical neural network using the generated image group for learning as input data.

(5) In this case, it is preferable that the generation step includes a process of horizontally reflecting the face image extracted from the sample image.
In this way, the number of learning image groups obtained from the same number of sample images can be doubled. Therefore, it is possible to save the trouble of collecting the sample image with the label. As a deeper cause, there is a problem that the deep learning classifier, which is often used for image processing, does not have inversion invariance. For this reason, under shooting conditions in different directions, the extracted object characteristics are different even for the same object, which causes a decrease in recognition accuracy. Therefore, the recognition accuracy can be improved by adding the horizontal reflection processing to the learning image of the face.

(6) The facial expression recognition device of the present invention is a device that recognizes facial facial expressions included in a captured image, and is for learning that has features related to at least three types of information including facial unevenness information, texture information, and contour information . A processing unit having a hierarchical neural network in which parameters are learned using an image group as input data, and a plurality of input images generated by extracting features related to at least the three types of information from the captured image. The recognition result output by the image generation unit that inputs the input image of the above to the processing unit and the hierarchical neural network that has learned the plurality of input images as input data is output to the outside as the facial expression of the captured image. It is provided with an output unit.

According to the expression recognition device of the present invention, the input data used for learning the parameters of the hierarchical neural network possessed by the processing unit has characteristics related to at least three types of information including facial unevenness information, texture information and contour information. It consists of a group of images for learning.
Further, the input data for facial expression recognition by the trained hierarchical neural network generated by the image generation unit is obtained from a plurality of input images generated by extracting at least the features related to the above three types of information from the captured image. Become.

Therefore, the accuracy of facial expression recognition using the hierarchical neural network is higher than that of the conventional technique in which no preprocessing is performed before input to the hierarchical neural network or preprocessing for extracting only a single feature factor is performed. Can be improved (see FIG. 8).

(7) The computer program of the present invention is a computer program for causing a computer device capable of executing image processing to execute a process of recognizing a facial expression included in a captured image, and the above-mentioned (1) to (1) to ( It includes the same steps as the facial expression recognition method of 5).
Therefore, the computer program of the present invention has the same effect as the facial expression recognition methods (1) to (5) described above.

(8) The advertisement management system of the present invention includes an advertisement display device, a photographing device for photographing a viewer of an advertisement image displayed by the advertisement display device, and a control device having the above-mentioned expression recognition device. The control device manages the recognition process of recognizing the facial expression of the visual viewer from the captured image including the visual visual photographed by the photographing device, the aggregation process of aggregating the recognition result of the facial expression, and the aggregation result of the advertisement. The presentation process to be presented to the person is executed.

According to the advertisement management system of the present invention, the control device recognizes the facial expression of the viewer from the captured image including the viewer captured by the photographing device, and aggregates the recognition result of the facial expression. Since the presentation process of presenting the result to the manager of the advertisement is executed, the manager can examine the significance of the current advertisement image from the presented aggregated result.
Therefore, the administrator can make a judgment such as canceling or continuing the advertisement based on the current advertisement image, or modifying the current advertisement image.

The present invention can be realized not only as a system and an apparatus having the above-mentioned characteristic configuration, but also as a computer program for causing a computer to execute such a characteristic configuration.
Further, the present invention described above can be realized as one or more semiconductor integrated circuits that realize a part or all of a system and an apparatus.

According to the present invention, the accuracy of facial expression recognition using a hierarchical neural network can be improved.

It is a block diagram of the image processing apparatus which concerns on embodiment of this invention. It is a schematic block diagram of CNN included in a CNN processing part. It is a conceptual diagram of the processing content of a convolutional layer. It is a conceptual diagram of the structure of the receptive field. It is explanatory drawing of the processing content of an image generation part. It is explanatory drawing which shows the specific example of the facial expression recognition method using an image processing apparatus. It is a structural drawing of the deep CNN constructed in the CNN processing part. It is a graph which compared the error rate when the input image is an AGE image, and the error rate when the input image is an RGB image. It is an overall block diagram of the advertisement management system of this embodiment. It is a bar graph which shows an example of the aggregation result of facial expression recognition.

Hereinafter, the details of the embodiment of the present invention will be described with reference to the drawings. In addition, at least a part of the embodiments described below may be arbitrarily combined.

[Overall configuration of image processing device]
FIG. 1 is a block diagram of an image processing device 1 according to an embodiment of the present invention.
As shown in FIG. 1, the image processing device 1 of the present embodiment includes, for example, a GPU (Graphics Processing Unit) mounted on a PC (Personal Computer) (not shown). The image processing device 1 includes an image generation unit 2, a CNN processing unit 3, a learning unit 4, and an output unit 5 as functional units realized by a computer program recorded in the memory of a PC.

The image generation unit 2 performs a process of extracting a predetermined feature from the labeled sample image 7 or the captured image 8 to be recognized, and performs an input image to the CNN processing unit 3 (hereinafter, also referred to as “input data”. ) Is generated. The image generation unit 2 inputs the input image to the CNN processing unit 3.
The CNN processing unit 3 executes recognition processing using CNN (facial expression recognition of a face image in this embodiment) on the input data, and learns the recognition result (specifically, the probability for each classification class). Input to unit 4 or output unit 5.

Specifically, when training the CNN using the labeled sample image 7, the CNN processing unit 3 inputs the recognition result to the learning unit 4.
On the other hand, when the trained CNN processing unit 3 specifies the classification class of the face image included in the captured image 8 (the type of facial expression of the face image in this embodiment), that is, the image processing device 1 operates as a facial expression classifier. In this case, the CNN processing unit 3 inputs the recognition result to the output unit 5.

The learning unit 4 updates the parameters (weights and biases) held by the CNN processing unit 3 based on the input recognition result, and stores the updated parameters in the CNN processing unit 3.
The output unit 5 identifies the classification class of the input image based on the input recognition result. Specifically, the classification class with the highest probability input from the CNN processing unit 3 is set as the classification class of the input image. The classification result output by the output unit 5 is displayed on the display of the PC or the like to notify the operator of the PC.

[Processing content of CNN processing unit]
(CNN configuration example)
FIG. 2 is a schematic configuration diagram of a CNN included in the CNN processing unit 3.
As shown in FIG. 2, the CNN constructed in the CNN processing unit 3 has three arithmetic processing layers: a convolutional layer (also referred to as a “downsampling layer”) C1 and C2, a pooling layer P1 and P2, and a fully connected layer F. And the final layer E, which is the output layer of the CNN.

The pooling layers P1 and P2 are arranged after the convolutional layers C1 and C2, and the fully connected layer F is arranged after the last pooling layer P2. The final layer E of the CNN includes the same number of final nodes (10 in FIG. 2) as the preset number of classification classes.
FIG. 2 illustrates a case where there are two convolutional layers C1 and C2 and corresponding pooling layers P1 and P2. However, the number of convolutional layers and pooling layers may be three or more. Further, at least one fully connected layer F is arranged.

The j-th node in a certain layer C1, P1, C2, P2 receives input x _i (i = 1, 2, ... m) from m nodes of the immediately preceding layer, respectively, and biases them to the weighted sum. The intermediate variable u _j is calculated by adding. That is, the intermediate variable u _j is calculated by the following equation. Note that in the following _{equation, w ij} is the weight, _{b j} is the bias.

The response y _j in which the intermediate variable u _j is applied to the activation function a (・), which is a non-linear function, that is, y _j = a (u _j ) becomes the output of the node of this layer, and this output is input to the next layer. Will be done.
For the activation function a, a "sigmoid function" or a (x _j ) = max (x _j , 0) is used. In particular, the latter activation function is called "ReLU (Rectified Linear Unit)". ReLU is often used in recent years because it contributes to good convergence and improvement of learning speed.

In the vicinity of the output layer of the CNN, one or more fully connected layers F in which all the nodes between adjacent layers are connected are arranged. The final layer E that gives the output of CNN is designed in the same way as a normal neural network.
When the purpose is to classify the input image, the same number of nodes as the number of classification classes are arranged in the final layer E, and the "softmax function" is used for the activation function a of the final layer E.

Specifically, the following equation is calculated based on the inputs u _j (j = 1, 2, ... n) to the n nodes. At the time of recognition, the index j = argmax _j p _{j of the} node in which p _j has the maximum value is selected as the estimation class.

(Processing content of convolutional layer)
FIG. 3 is a conceptual diagram of the processing contents of the convolutional layers C1 and C2.
As shown in FIG. 3, the inputs of the convolutional layers C1 and C2 are in the form of N pixels (N channels) having a vertically long size of S × S pixels.
Hereinafter, the image of this format will be described as S × S × N. Further, the input of S × S × N is described as x _ijk (where (i, j, k) ∈ [0, S-1], [0, S-1], [1, N]).

In CNN, the number of channels of the first input layer (convolutional layer C1) is N = 1 if the input image is grayscale, and N = 3 (3 channels of RGB) if the input image is color.
In the convolutional layers C1 and C2, the calculation of convolving a filter (also referred to as "kernel") into the input x _ijk is executed.

This calculation includes filter convolution in general image processing, for example, a process of two-dimensionally convolving a small-sized image into an input image to blur the image (Gaussian filter), and a process of emphasizing edges (sharpening filter). ) Is basically the same process.
Specifically, a two-dimensional filter having an L × L size is convoluted into pixels having an input x _ijk size S × S of each channel k (k = 1 to N), and the result is obtained from all channels k = 1 to N. Add over. The calculation result is in the form of a 1-channel image _uij .

If the filter is defined as w _ijk (where (i, j, k) ∈ [1, L-1], [1, L-1], [1, N]), u _ij is calculated by the following equation. ..

However, _Pij is a square region of size L × L pixels having pixels (i, j) in the image as vertices. That is, _Pij is defined by the following equation.

b _k is a bias. In this embodiment, the bias is common to all output nodes for each channel. That is, b _ijk = b _k .
The filter may be applied at intervals of multiple pixels instead of all pixels. That is, for a given number of pixels _s, define a _{P ij} as _{follows, w p-i, q-} j, a _{k w p-si, q-} sj, to calculate the _{u ij} replaced with _k May be good. This pixel spacing s is called a "slide".

The u _ij calculated as described above then passes through the activation function a (.) And becomes the output y _ij of the convolutional layers C1 and C2. That is, y _ij = a (u _ij ).
As a result, for one filter w _ijk , the output y _ij for one channel of S × S having the same vertical and horizontal sizes as the input x _ijk can be obtained.

If N'similar filters are prepared and the above calculations are performed independently for each, the output of S × S for N'channels, that is, the output of S × S × _N'size y _ijk (however, (i, j, k) ∈ [1, S-1], [1, S-1], [1, N']) is obtained.
The output y _{ijk for} this _N'channel becomes the input x _ijk to the next layer. FIG. 3 shows the calculation contents for one of the N'filters.

The above calculation can be expressed as a single-layer network in which interlayer nodes are connected in a special form, as shown in FIG. 4, for example. FIG. 4 is a conceptual diagram of the structure of the receptive field. In the figure on the left, the receptive field is represented by a rectangle, and in the figure on the right, the receptive field is represented by a node.
Specifically, each node in the upper layer is connected to a part of each node in the lower layer (this is called a "local receptive field"). In addition, the join weight is common among each node (this is called "weight sharing").

(Processing content of pooling layer)
As shown in FIG. 2, the pooling layers P1 and P2 exist in pairs with the convolutional layers C1 and C2. Therefore, the outputs of the convolutional layers C1 and C2 are the inputs to the pooling layers P1 and P2, and the inputs of the pooling layers P1 and P2 are in the form of S × S × N.
The purpose of the pooling layers P1 and P2 is to discard some information about the position of the image where the filter response was strong, and to realize the invariance of the response to a slight change in the characteristics.

The nodes (i, j) of the pooling layers P1 and P2 have local receptive fields _{Pi and j} in the input side layer, similarly to the convolutional layers C1 and C2. Node pooling layer P1, P2 (i, j) is to aggregate local receptive field _{P i,} the internal nodes of the _{j (p, q) ∈P i} , the output _{y p} of the _{j, q} to one.
The sizes of the local receptive fields _{Pi and j} of the pooling layers P1 and P2 are set independently of those of the convolutional layers C1 and C2 (filter size).

When the input is a plurality of channels, the above processing is performed for each channel. That is, the number of output channels of the convolutional layers C1 and C2 and the pooling layers P1 and P2 are the same.
The pooling is performed by thinning out the images in the vertical and horizontal directions (i, j). That is, two or more strides are set. For example, when s = 2, vertical and horizontal size of the output becomes half the vertical and horizontal size of the input, the number of output nodes of the pooling layer becomes 1 / s ² times the number of input nodes.

There are "average pooling" and "maximum pooling" as a method of collecting the inputs from the nodes inside the receptive fields _{Pi and j} into one.
The average pooling is a method of outputting the average value of the input x _pqk from the nodes belonging to _{Pi and j as shown} in the following equation.

Maximum pooling is a method of outputting the maximum value of the input x _pqk from the nodes belonging to _{Pi and j as shown} in the following equation. Average pooling was predominant in CNN's early studies, but maximum pooling is now commonly adopted.

Unlike the convolutional layers C1 and C2, the pooling layers P1 and P2 do not have weights that change due to learning, and the activation function is not applied.
In the CNN of the present embodiment, either the average pooling or the maximum pooling may be adopted, but the maximum pooling is adopted in the implementation example of the CNN shown in FIG. 7.

[Processing content of the learning department]
"Supervised learning" is the basis of CNN training. Also in this embodiment, the learning unit 4 executes supervised learning.
Specifically, the learning unit 4 is executed by minimizing the classification error of each sample image for a set including a large number of labeled sample images as learning data. This process will be described below.

Each node of the final layer E of the CNN processing unit 3 outputs the probability _pj (j = 1, 2, ... n) for the corresponding class by normalization by the softmax function ([Equation 2] described above). .. This probability p _j is input to the learning unit 4.
Learning unit 4, so as to minimize the classification error calculated from the input probability p _j, it updates the parameters such weight set in the CNN processing unit 3.

Specifically, the learning unit 4 has ideal outputs d1, d2, ... dn (label) for the input sample and output p1. p2. The dissociation of pn is calculated by the cross entropy C of the following equation. This cross entropy C is the classification error.

The target outputs d1, d2, ... dn are set so that d _j = 1 only in the correct answer class j and d _k = 0 in all other k (≠ j).
Learning unit 4, as in the above cross-entropy C decreases, the bias b _k coefficients w _ijk each node of the filter of each convolution layers C1, _C2, and total binding layer disposed on the output layer side of the CNN Adjust the weight and bias of F.

A stochastic gradient descent method is used to minimize the classification error C. Learning unit 4, the error gradient related weights and biases the (∂C / ∂w _ij), calculated by backpropagation (BP method). The calculation method by the BP method is the same as that of a normal neural network.
However, in the back propagation when the CNN processing unit 3 adopts the maximum pooling, the value of which node in the pooling region is selected at the time of forward propagation for the training sample is memorized, and only that node is combined at the time of back propagation ( Combine with weight 1).

The evaluation of the classification error C by the learning unit 4 and the update of the parameters (weights, etc.) based on the evaluation may be performed for all the learning samples. However, from the viewpoint of convergence and calculation speed, it is preferable to execute each set (mini-batch) of several to several hundred samples. The update amount Δw _ij of the weight w _{ij in} this case is determined by the following equation.

In the above equation, Δw _ij ^(t) is the current weight update amount, and Δw _ij ^(t-1) is the previous weight update amount. The first term of the above equation is a term representing the amount of correction of _wij for reducing the error by the gradient descent method, and ε is the learning rate.
The second term of the above equation is momentum. Momentum suppresses weight bias due to mini-patch selection by adding α (~ 0.9) times the previous update amount. The third term is weight decay. Weight attenuation is a parameter that prevents the weight from becoming excessive. The same applies to the update of the bias b _k .

[Processing content of image generator]
FIG. 5 is an explanatory diagram of the processing content of the image generation unit 2.
As shown in FIG. 5, the image processing executed by the image generation unit 2 includes two processes, a “face extraction process” and a “feature extraction process”.

The face extraction process is a process of trimming a rectangular image (original image of a face), which is mostly a human face portion, from a source image such as a sample image 7 or a captured image 8 (see FIG. 1).
The feature extraction process is a process of generating an input image to be supplied to the CNN processing unit 3 by highlighting a predetermined feature in the rectangular image obtained by the face extraction process.

In the feature extraction process of the present embodiment, a total of three types of input image groups are generated by extracting three types of features of "angle", "gradient", and "edge" from the rectangular image. Will be done. Hereinafter, these three types of input images are also referred to as "AGE images".
Here, assuming that the two-dimensional coordinates of the pixel points of the rectangular image are (x, y) and the pixel value (for example, RGB value) of each pixel point (x, y) is "I", the angle A, the gradient G, and the contour The mathematical meanings of E are as follows.

Angle A: “Direction angle” of the gradient vector ∇f = (∂I / ∂x, ∂I / ∂y) of the pixel value I at each pixel point (x, y)
Gradient G: The "norm" (length) of the gradient vector ∇f = (∂I / ∂x, ∂I / ∂y) of the pixel value I at each pixel point (x, y).
Contour E: Position information (edge image) of pixel points (x, y) at which the pixel value I changes sharply.

The angle A represents geographical information (hereinafter, referred to as “unevenness information”) such as unevenness inside the face included in the rectangular image.
The gradient G represents information on the texture of the skin, hair, etc. inside the face included in the rectangular image (hereinafter, referred to as “texture information”).
The contour E represents an outline (hereinafter, referred to as “contour information”) of each part such as the head, eyes, mouth, and nose of the face included in the rectangular image.

The values of the three feature quantities (angle A, gradient G, and contour E) at each pixel point (x, y) are v1, v2, v3, respectively, and the pixel values I to v1, v2 of the rectangular image at each pixel point. Assuming that the filters for generating v3 are Da, Dg, and De, respectively, the following equation holds.

In this case, the calculation formulas for the filters Da and Dg are as follows.

The calculation formula of the filter De is as follows, for example.

In order to clearly distinguish the contour from the information around it, the shading value (black and white value) of each direction point around the contour point (x, y) is the contour point (x, y) expressed by the following equation. It is desirable to adjust according to the angle value θ _e .

For example, when θ _e (x, y) = 0, the point (x, y) has a vertical contour, so by darkening the left side of the contour information extracted from that point, the left side is darkened. The outline of the object can be clearly expressed.
As described above, each of the filters Da, Dg, and De brings the features of the angle A, the gradient G, and the contour E at each pixel point (x, y) as an extracted AGE image. In the feature extraction process, all the pixel points (x, y) of the rectangular image are scanned once by the above filter, so that three inputs including information on the angle A, the gradient G, and the contour E are input from one rectangular image. An image is generated.

Other image processing executed by the image generation unit 2 includes a process of changing the size of the rectangular image trimmed by the face extraction process, a process of generating horizontal reflection (reflection) of the rectangular image, and the like. May be good.

The above image processing executed by the image generation unit 2 can be executed by open source software such as "VLfeat", "OpenCV", "ImageStone", "GIMP", and "CxImage".
The filters Da and Dg can be calculated from Ix and Iy obtained by a partial differential filter such as VLfeat or OpenCV. Further, as the filter De, an OpenCV Sobel filter, a Laplace filter, a Canny filter and the like can be used.

[Specific example of facial expression recognition method]
FIG. 6 is an explanatory diagram showing a specific example of a facial expression recognition method using the image processing device 1.
As shown in FIG. 6, the facial expression recognition method of the present embodiment is roughly divided into two steps, a “learning step” and a “recognition step”.
The learning step is a step of learning the CNN of the image processing apparatus 1 using a plurality of sample images 7. The recognition step is a step of causing the CNN processing unit 3 including the learned CNN to recognize the facial expression of the face image included in the captured image 8.

In the learning step, a plurality of sample images 7 (raw images with labels) are changed (trimmed) into face images having a size of 64 × 64. N in FIG. 6 represents the number of images for training in CNN.
Next, in order to increase the number of images from N to G, horizontal reflection (reflection) is applied to N images of size 64 × 64, and patches of size 56 × 56 are extracted respectively. In addition, G = 2 × N.

Next, three types of input data (AGE image group in the present embodiment) obtained by extracting facial unevenness information, texture information, and contour information from G-sheet size 56 × 56 patches are generated. That is, from G patches, AGE images having a size of 56 × 56 and 3 × G are generated. The above processing is executed by the image generation unit 2 of the image processing device 1.

The AGE images having a size of 56 × 56 and 3 × G (learning image groups input to each CNN) are input to the CNN processing unit 3 of the image processing device 1 in order to train the convolutional network.
In this training, the learning unit 4 adjusts parameters such as weights and biases for the CNN processing unit 3.

In the recognition step, the captured image 8 (raw image without a label), which is the target of facial expression recognition, is changed (trimmed) to a face image having a size of 56 × 56.
Next, three types of input data (AGE image in the present embodiment) are generated by extracting the unevenness information, the texture information, and the contour information of the face from one face image having a size of 56 × 56. The above processing is executed by the image generation unit 2 of the image processing device 1.

The three AGE images having a size of 56 × 56 are input to the CNN processing unit 3 of the image processing device 1 for facial expression recognition of the facial image.
In this facial expression recognition, the CNN processing unit 3 identifies a preset facial expression classification class for the input AGE image by using the CNN having the learned parameters. The specified classification class is input to the output unit 5. The output unit 5 displays the input classification class on a display of a PC or the like.

[Recommended CNN structure example]
FIG. 7 is a structural diagram of a deep CNN constructed in the CNN processing unit 3.
As shown in FIG. 7, the CNN architecture for human facial expression recognition recommended by the inventors of the present application is a stack of convolution layers C1 to C4 for converting an input volume into an output volume and fully coupled layers A1 to A3. It is composed of the body.

Each layer C1-C4, A1-A3 of the CNN has neurons arranged three-dimensionally in width, height and depth.
The width, height and depth of the first input layer C1 are preferably 56 × 56 × 3. The neurons inside the convolutional layers C2 to C4 and the fully connected layer A1 are connected only to the nodes in a small area called the receptive field of the previous layer.

The spatial size of the output volume can be calculated by the following equation.
W2 = 1 + (W1-K + 2P) / S
In the above equation, W1 is the size of the input volume. K is the field size of the nucleus (node) of the neuron in the convolutional layer. S means stride, that is, the distance between the centers of the receptive fields of adjacent neurons in the kernel map. P means the amount of zero padding used on the border.

In the CNN of FIG. 7, W1 = 56, K = 5, S = 2, P = 2 in the first convolutional layer C1. Therefore, the spatial size of the output volume of the second convolutional layer C2 is W2 = 1 + (56-5 + 2 × 2) / 2 = 28.5 → 28.
The network of FIG. 7 includes seven layers with weights. The first four are convolutional layers C1 to C4 and the remaining three are fully connected fully connected layers A1 to A3. Fully bonded layers A1 to A3 include dropouts.

The output of the last fully connected layer A3 is fed to 7-way SOFTMAX, which produces a distribution of 7 class labels, which is the final layer fully connected to this layer A3.
The neurons of the convolutional layers C2 to C4 and the fully connected layer A1 are connected to the receptive field of the previous layer, and the neurons of the fully connected layers A2 to A3 are connected to all the neurons of the previous layer.

The convolutional layers C1 and C2 are followed by a batch regularization layer. Each batch regularization layer is followed by a pooling layer that performs the maximal pooling described above.
The nonlinear mapping function for the convolutional layers C1 to C4 and the fully coupled layers A1 to A3 consists of a rectifying linear unit (ReLU).

The first convolutional layer C1 filters a 56 × 56 × 3 input image (AGE image) with a 2-pixel stride by 64 kernels having a size of 5 × 5 × 3.
Stride is the distance between the centers of the receptive fields of adjacent neurons in the kernel map. The stride is set to 1 pixel in all convolutional layers.

The input of the second convolutional layer C2 is the output of the first convolutional layer C1 batch-normalized and maximally pooled. The second convolution layer C2 filters the inputs with 128 kernels of size 3x3x64.
The third convolutional layer C3 has 128 kernels of size 3x3x64, which are connected to the output of the second layer C2 (batch normalization and MAX pooling).

The fourth convolutional layer C4 comprises 128 kernels having a size of 3 × 3 × 128. Fully connected fully connected layers A1 to A3 each include 1024 neurons.

[Recommended learning example]
The inventors of the present application actually trained (learned) the deep CNN of the structure of FIG. The training was conducted using "MATLAB", an open source numerical analysis software, for a PC equipped with an NVIDIA GTX745 4GB GPU.
In the learning step of CNN, there are important settings such as weight attenuation, momentum, batch size, learning rate and parameters including learning cycle. This point will be described below.

In the training by the inventors of the present application, an asynchronous stochastic gradient descent method having a momentum of 0.9 and a weight attenuation of 0.0005 was adopted. The following equation is the update rule for the weight w adopted this time.

In the above equation, i is the number of iterations and m is the momentum variable. ε means the learning rate. The third term on the right side is the average value of the correction amount of the weight w for reducing the error L in wi with respect to the i-th batch Di.
Increasing the batch size provides a more reliable gradient estimate and can reduce the learning time, but it does not provide the maximum stable increase in the learning rate ε. Therefore, it is necessary to select a batch size suitable for the CNN model.

Here, the results of training (learning) using batch sizes of 64, 128, 256, and 512 were compared for the convolutional layers C1 to C4, respectively. As a result, it was found that the batch size of 256 was optimal for the CNN of FIG.
Equal learning rates were used for all layers and manually adjusted throughout the training. The learning rate was initialized to 0.1, and when the error rate stopped improving at the current learning rate, the learning rate was divided by 10. In the training, an input image consisting of an AGE image was input, and the network was trained in about 20 cycles.

[Experimental example: Effect when AGE image is used as input image]
The inventors of the present application conducted an experiment to confirm the accuracy of facial expression recognition when an AGE image was used as an input image using a database of SFW (Static Facial Expression in the Wild) for CNN in FIG.

The input image SFW is assigned one of seven emotion labels of "calm", "joy", "anger", "surprise", "discomfort", "sadness", and "dislike".
Therefore, the emotion label output by the learned CNN is also one of the above seven types.

FIG. 8 is a graph comparing the error rate when the input image is an AGE image and the error rate when the input image is an RGB image. In FIG. 8, the horizontal axis represents the number of training cycles, and the vertical axis represents the error rate in each cycle.
The error rate means the probability that facial expression recognition fails. For example, an error rate of 0.6 means that when 10 facial expressions are recognized, 6 people fail and 4 people succeed. In the current facial expression recognition by deep CNN, there is only one with an error rate of about 0.6.

As shown in FIG. 8, when the input image is an RGB image, the error rate is about 0.65 in the case of 20 cycles. When the input image is an AGE image, the error rate is less than 0.6 after 10 cycles or more.
As is clear from the graph of FIG. 8, in the facial expression recognition using the deep CNN, if the AGE image is adopted as the input image, the discriminating power of the facial expression recognition is improved, and the conventional raw data (RGB image) is used as the input image. Compared with the case, the performance of facial expression recognition is significantly improved.

[Application example of image processing device]
FIG. 9 is an overall configuration diagram of the advertisement management system 10 of the present embodiment.
The advertisement management system 10 of the present embodiment is a management system that uses an image processing device 1 (see FIG. 1) capable of performing facial expression recognition of a face image included in a captured image for advertisement evaluation.

As shown in FIG. 9, the advertisement management system 10 includes an advertisement display device 11, a photographing device 12, an advertisement control device 13, and a management device 14.
The advertisement display device 11 includes, for example, an LED electric display board, a liquid crystal display, or the like. The advertisement display device 11 displays a predetermined advertisement image received from the advertisement control device 13 on the display surface. The advertisement image may be either a still image or a moving image. The advertisement display device 11 may be an advertisement signboard on which an advertisement poster is attached.

The photographing device 12 includes, for example, a digital camera that generates a digital image using a CCD (charge coupling element). The photographing device 12 is attached to the upper end portion of the advertisement display device 11 or the like, and photographs a person (hereinafter, referred to as “visual viewer”) who stands in front of the advertisement display device 11 and visually observes the advertisement.
The photographing device 12 transmits a photographed image including the face of the viewer to the advertisement control device 13. The captured image may be either a still image or a moving image.

The advertisement control device 13 includes a computer device that controls the advertisement display device 11 and the photographing device 12. The advertisement control device 13 includes a first communication unit 16, a second communication unit 17, a control unit 18, and a storage unit 19.
The first communication unit 16 includes a communication device that communicates with the advertisement display device 11 and the photographing device 12 according to a predetermined I / O interface standard. The communication between the first communication unit 16 and the advertisement display device 11 and the photographing device 12 may be either wired communication or wireless communication.

The second communication unit 17 includes a communication device that communicates with the management device 14 according to a predetermined communication standard such as a wired or wireless LAN.
The second communication unit 17 may communicate with the management device 14 via a public communication network such as the Internet (in the case of FIG. 9), or may communicate with the management device 14 only via the premises communication network. However, it may communicate directly with the management device 14. The communication between the second communication unit 17 and the management device 14 may be either wired communication or wireless communication.

The control unit 18 includes one or a plurality of CPUs (Central Processing Units) and a control device including the GPU (image processing unit 1 of FIG. 1) of the present embodiment described above.
The storage unit 19 includes a storage device including one or a plurality of RAMs (Random Access Memory) and a memory such as a ROM (Read Only Memory). The storage unit 19 functions as a temporary or non-temporary recording medium for various computer programs executed by the control unit 18, various data received from the management device 14, and the like.

In this way, the advertisement control device 13 is configured to include a computer. Therefore, each function of the advertisement control device 13 is exhibited by executing the computer program stored in the storage device of the computer by the CPU and GPU of the computer.
Such a computer program can be stored in a temporary or non-temporary recording medium such as a CD-ROM or a USB memory.

The control unit 18 reads and executes the computer program stored in the storage unit 19, thereby controlling communication with the first and second communication units 16 and 17, and various types useful for the administrator who operates the management device 14. The application can be realized.
For example, the control unit 18 sets the first communication unit 16 so that when the second communication unit 17 receives the advertisement image transmitted by the management device 14 to its own station, the control unit 18 transmits the received advertisement image to the advertisement display device 11. Control. After that, the advertisement display device 11 displays the received advertisement image on the display surface.

When the first communication unit 16 receives the photographed image transmitted by the photographing device 12, the control unit 18 executes facial expression recognition on the face image included in the received photographed image, and sends the facial expression classification result to the management device 14. The second communication unit 17 is controlled so as to transmit.
The storage unit 19 stores a CNN having a predetermined structure (for example, FIG. 7) capable of performing facial expression recognition of a facial image, learned weights and biases with respect to the CNN, and the like. The GPU of the control unit 18 executes facial expression recognition for the viewer's face image included in the captured image by the learned CNN stored in the storage unit 19.

The management device 14 comprises, for example, a server computer device operated by the advertisement manager. In the example of FIG. 9, only one advertisement control device 13 is connected to the management device 14, but a plurality of advertisement control devices 13 may be connected to the management device 14.
The management device 14 can execute the distribution processing of the advertisement image to one or a plurality of advertisement control devices 13. Specifically, the management device 14 transmits the advertisement image input by the administrator to the predetermined advertisement display device 10. Therefore, the advertisement image of the advertisement display device 11 can be switched.

The management device 14 can execute the aggregation processing of the classification results received from one or a plurality of advertisement control devices 13. Specifically, the management device 14 aggregates a large number of classification results received from the advertisement control device 13 during a predetermined period. The management device 14 presents the total result to the manager by graphing or tabulating the total result and displaying it on the display.
Therefore, the administrator can determine the evaluation of the significance of the advertisement image, the continuation or cancellation of the display of the advertisement image, the modification of the advertisement image, and the like based on the aggregation result presented by the management device 14.

FIG. 10 is a bar graph showing an example of the aggregated results of facial expression recognition.
In the bar graph of FIG. 10, the horizontal axis is a classification class of seven types (“calm”, “joy”, “anger”, “surprise”, “discomfort”, “sadness”, and “dislike”) related to human facial expressions. .. The vertical axis is the occurrence rate of the seven types of classification classes.

The bar graph in the upper part of FIG. 10 shows the aggregation result that leads to the “continuation” of the advertisement image.
In this tabulation result, the ratio of "joy" is higher than that of other classification classes, so it can be estimated that many viewers are happy to see the current advertisement image.
Therefore, when the aggregated result as shown in the bar graph in the upper part of FIG. 10 is obtained, it is considered that the administrator should judge the continuation of the advertisement by the current advertisement image.

The bar graph at the bottom of FIG. 10 shows the aggregated results leading to "cancellation" and "modification" of the advertisement image.
In this tabulation result, the ratios of "discomfort" and "dislike" are higher than those of other classification classes, so it can be estimated that many viewers are discomforted by seeing the current advertisement image.
Therefore, when the aggregated result as shown in the bar graph at the bottom of FIG. 10 is obtained, the administrator should decide to stop the advertisement by the current advertisement image or to modify the current advertisement image. Be done.

In the example of FIG. 9, the advertisement control device 13 executes facial expression recognition and transmits the classification result to the management device 14, and the management device 14 aggregates the classification results. However, the advertisement control device 13 aggregates the classification results. Is executed, and the total result may be transmitted to the management device 14.
Further, the advertisement control device 13 may transfer the captured image to the management device 14, and the management device 14 may execute the facial expression recognition of the face image included in the captured image, and the classification and aggregation thereof. Further, the advertisement control device 13 and the management device 14 may be composed of a control device having the same housing including one computer device.

The steps of the advertisement management method realized by the above-mentioned advertisement management system 10 are as follows.
Step 1) During the period of displaying a predetermined advertisement image (still image or moving image) on the advertisement display device 11, the viewer in front of the advertisement display device 11 is photographed by the camera 12.

Step 2) The facial expression recognition of the face image included in the captured image is executed by a computer device such as the advertisement control device 13 and the management device 14, and the classification results are totaled. Specifically, the total number of recognized facial images is used as the denominator, and the ratio of each classified facial expression is obtained.
Process 3) The manager determines whether to continue, cancel, or modify the current advertising image based on the aggregated result.

[Other variants]
The embodiments disclosed this time (including modified examples) are examples in all respects and are not restrictive. The scope of rights of the present invention is not limited to the above-described embodiment, and includes all modifications within a range equivalent to the configuration described in the claims.
For example, in the above-described embodiment, three types of input images (AGE images) are generated by extracting three types of features from the original image, but four or more types of features including the three types of features are generated. It may be extracted and four or more kinds of input images may be generated.

In the above embodiment, the neural network is composed of a convolutional neural network (CNN), but it may be a hierarchical neural network having another structure that does not have a convolutional layer.
In the above-described embodiment, the control unit 18 of the advertisement control device 13 may execute the facial expression recognition by an algorithm other than the deep CNN, as long as it can accurately recognize the facial expression of the facial image.

1 Image processing device 2 Image generation unit 3 CNN processing unit (processing unit)
4 Learning unit 5 Output unit 7 Sample image 8 Captured image 10 Advertisement management system 11 Advertisement display device 12 Shooting device 13 Advertisement control device (control device)
14 Management device (control device)
16 1st communication unit 17 2nd communication unit 18 Control unit 19 Storage unit

Claims (8)

It is a method of recognizing facial expressions included in captured images.
A learning step in which a hierarchical neural network learns parameters using a learning image group having features related to at least three types of information including facial unevenness information, texture information, and contour information as input data.
A plurality of input images have been generated by extracting features related to at least the three types of information from the captured images, and the generated facial expressions included in the captured images have been learned using the generated plurality of input images as input data. A facial expression recognition method including a recognition step for causing the hierarchical neural network to recognize. The facial expression recognition method according to claim 1, wherein the hierarchical neural network is a convolutional neural network. The unevenness information is the direction angle of the gradient vector of the pixel value at each pixel point.
The texture information is the norm of the gradient vector of the pixel values at each pixel point.
The facial expression recognition method according to claim 1 or 2, wherein the contour information is position information of a pixel point whose pixel value changes sharply. The learning step includes a generation step of generating the learning image group by extracting features related to at least the three types of information from a plurality of sample images.
Any one of claims 1 to 3, including an update step of updating the parameters of the network based on the recognition result output by the hierarchical neural network using the generated image group for learning as input data. The facial expression recognition method described in the section. The facial expression recognition method according to claim 4, wherein the generation step includes a process of applying horizontal reflection to a face image extracted from the sample image. It is a device that recognizes facial expressions included in captured images.
A processing unit having a hierarchical neural network in which parameters are learned by using a learning image group having features related to at least three types of information including facial unevenness information, texture information, and contour information as input data.
An image generation unit that extracts features related to at least the three types of information from the captured image to generate a plurality of input images, and inputs the generated plurality of input images to the processing unit.
A facial expression recognition device including an output unit that outputs a recognition result output by the hierarchical neural network that has been trained using the plurality of input images as input data to the outside as a facial expression of the captured image. A computer program for causing a computer device capable of performing image processing to perform processing for recognizing facial expressions included in a captured image.
A learning step in which a hierarchical neural network learns parameters using a learning image group having features related to at least three types of information including facial unevenness information, texture information, and contour information as input data.
Features related to at least the three types of information are extracted from the captured image to generate a plurality of input images, and the generated plurality of input images are used as input data to learn facial expressions included in the captured image. A computer program including a recognition step to be recognized by the hierarchical neural network. Advertising display device and
A photographing device that captures a viewer of an advertisement image displayed by the advertisement display device, and
An advertisement management system including a control device having the facial expression recognition device according to claim 6.
The control device has a recognition process for recognizing the facial expression of the viewer from a photographed image including the viewer taken by the photographing device, an aggregation process for aggregating the recognition result of the facial expression, and an advertisement manager for the aggregation result. An ad management system that executes the presentation process presented to.