JP2022109742A - Program, device, and method for estimating label by using input intermediate layer different for every area image of object - Google Patents

Abstract

To provide a program, a device, and a method for analyzing every area image of an object from an object image and estimating a label that comprehensively evaluates the area images.SOLUTION: A program causes a face expression recognition device (computer) to function as area division means that divides an object image into area images according to different area types, input intermediate layers that are each provided for every area type, receive input of the area images, and output intermediate feature quantities by using a model trained in advance, an intermediate feature quantity fusion layer that fuses the intermediate feature quantities output from the plurality of input intermediate layers to output a fused intermediate feature quantity, and an output layer that receives input of the fused intermediate feature quantity and estimates a label by using the model trained in advance. The intermediate feature quantity fusion layer fuses the plurality of intermediate feature quantities with weighting different for every input intermediate layer. The object image is an image capturing the face of a person who wears an attachment material, and the label is the facial expression of the person. The program further has face area detection means that receives input of the object image and detects the person's face area image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a machine learning engine technique for estimating comprehensively evaluated labels from images in which multiple objects are captured. In particular, it is suitable for estimating the facial expression from a face image of a person wearing a wearable object such as a mask.

Machine learning engine technology that recognizes people and objects from captured images is developing. In particular, the accuracy of face recognition for recognizing a person from a face image is rapidly improving with the development of deep learning technology. For example, facebook announced that the accuracy of DeepFace (registered trademark), a face recognition technology using deep learning, has reached 97.35% (for example, see Non-Patent Document 1).
Also, in order to train a learning model for a machine learning engine, it is necessary to use a large amount of teacher images. , has realized emotion recognition technology (see, for example, Non-Patent Document 2).

Conventionally, there is a technology in which feature amounts of a large number of face images are learned in advance for each emotion, and the emotion is recognized from the face image (for example, see Patent Document 1). Specifically, there are Ekman 7-classified facial expression models (neutral, joy, disgust, anger, surprise, sadness, and fear) and 3-classified emotion models of positive, negative, and neutral.

There is also a technique of having a recognizer for each of a plurality of recognition modes based on the state of a target person, and applying any one of the recognizers according to the recognition mode during face recognition (see Patent Document 2, for example). The condition of the target person's face includes whether or not he or she is wearing a mask, glasses, sunglasses, a hat, or the like. According to this technique, a recognition mode is selected from the closed area of the target person's face, and a recognizer based on the recognition mode determines success or failure of authentication. That is, each recognizer was trained from teacher images with different closed regions.

Furthermore, there is also a technology of a face recognition engine "specialized for wearing a mask" that extracts and matches feature points around eyes not covered by a mask (see, for example, Non-Patent Document 3). According to this technology, 1:1 authentication when wearing a mask achieved an authentication rate of 99.9% or more.

Furthermore, there is also a facial expression recognition AI (Artificial Intelligence) technology developed by the applicant of the present application (see, for example, Non-Patent Document 4). Face recognition technology is generally used for unlocking by successful authentication. On the other hand, applications of facial expression recognition technology include an automatic photographing function based on smile detection, and a marketing survey of receptivity based on facial expression analysis of TV program viewers.

JP 2011-150381 A JP 2018-165983 A

Taigman, Yaniv, et al. "Deepface: Closing the gap to human-level performance in face verification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2014. affectiva, [online], [searched on January 4, 2021], Internet <URL: https://affectiva.jp/reason.html> “NEC Develops Face Recognition Engine Specializing in Wearing a Mask--Recognition Rate is 99.9% or More”, [online], [Searched on January 4, 2021], Internet <URL: https://japan. cnet.com/article/35160036/> "Angle-free facial expression recognition AI", [online], [searched on January 4, 2018], Internet <URL: https://www.kddi-research.jp/newsrelease/2018/080201.html> "Understanding object detection and segmentation with Mask R-CNN", [online], [searched on January 4, 2021], Internet <URL: https://qiita.com/shtmr/items/ 4283c851bc3d9721ed96>

In recent years, since the outbreak of the new coronavirus infection, it has become common to wear masks and goggles on the face. When such a wearable is worn on the face, it covers up to 70% of the face area. For this reason, a problem has arisen that faces and expressions cannot be sufficiently recognized. According to general face recognition algorithms, it is necessary to capture as many features as possible, such as eyes, nose, mouth, cheeks, and facial muscles, from a face image.
For example, according to the technology described in Non-Patent Document 3, a machine learning engine for face recognition is trained with a large number of teacher images of faces and facial expressions wearing masks and goggles.

However, in the case of facial expression recognition, for example, wearing a mask on the face makes it impossible to extract feature amounts from most of the nose/mouth and most of the muscles of the cheeks/face. As a result, the recognition accuracy of facial expressions is greatly reduced. In addition, facial expression recognition technology differs from face recognition technology in that it authenticates individuals on a 1:1 basis. There was also the problem of not being able to

On the other hand, the inventors of the present application have investigated the possibility of estimating a facial expression by separately analyzing an exposed face area and other areas from a person's face image and evaluating them comprehensively. thought. We thought that this would require a technique to analyze each area image of the object separately from the target image and estimate the label by comprehensively evaluating them.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a program, an apparatus, and a method for analyzing each area image of an object separately from a target image and estimating a label obtained by comprehensively evaluating them.

According to the present invention, in a program that causes a computer to estimate a label from a target image,
area dividing means for dividing a target image into area images of different area types;
a plurality of input intermediate layers provided for each area type, inputting an area image, and outputting an intermediate feature amount using a pre-trained model;
an intermediate feature value fusion layer that fuses intermediate feature values output from a plurality of input intermediate layers and outputs a fused intermediate feature value;
It is characterized by having a computer function as an output layer that inputs fused intermediate feature values and estimates labels using pre-trained models.

According to another embodiment of the program of the present invention,
It is also preferable that the intermediate feature value fusion layer causes a computer to fuse a plurality of intermediate feature values with different weightings for each input intermediate layer.

According to another embodiment of the program of the present invention,
The target image is an image in which the face of the person wearing the attachment is reflected,
A label is a person's facial expression,
further comprising face area detection means for inputting a target image and detecting a person's face area image;
It is also preferable that the region dividing means causes a computer to function so as to input a face region image of a person and divide it into a face-exposed region image and a wearable object region image as region images of different region types.

According to another embodiment of the program of the present invention,
It is also preferred that the wearable be a mask, eyeglasses, goggles, or sunglasses for the computer to function.

According to another embodiment of the program of the present invention,
Using a teacher image with a teacher label,
The area dividing means divides the teacher image into area images of different area types,
The input hidden layer consists of an input layer and a hidden layer in the neural network, and has a model trained so that each region image based on the teacher image is input and the teacher label is output from the output layer,
The intermediate feature value fusion layer is trained to input the intermediate feature value output from each input intermediate layer and output a teacher label from the output layer, and derives different weights for each input intermediate layer,
The output layer also preferably functions a computer with a model trained to input fused intermediate features based on the training image and output training labels.

According to another embodiment of the program of the present invention,
It is also preferable that the area dividing means estimates an area type for each pixel of the input image and causes the computer to detect the area image based on the boundary line (segmentation) of the area image.

According to the present invention, in an estimation device that estimates a label from a target image,
area dividing means for dividing a target image into area images of different area types;
a plurality of input intermediate layers provided for each area type, inputting an area image, and outputting an intermediate feature amount using a pre-trained model;
an intermediate feature value fusion layer that fuses intermediate feature values output from a plurality of input intermediate layers and outputs a fused intermediate feature value;
It is characterized by having an output layer that inputs the fused intermediate feature amount and estimates the label using a pre-trained model.

According to the present invention, an estimation method for a device for estimating a label from a target image includes:
The device
a first step of dividing the target image into area images of different area types;
a second step of inputting an area image using an input intermediate layer provided for each area type and outputting an intermediate feature amount using a pre-trained model;
a third step of fusing intermediate feature values output from a plurality of input intermediate layers to output a fused intermediate feature value;
and a fourth step of inputting the fused intermediate feature amount and estimating the label using a pre-trained model.

According to the program, apparatus and method of the present invention, it is possible to analyze each area image of the object separately from the target image and estimate the label by comprehensively evaluating them.

FIG. 3 is a functional configuration diagram of the facial expression recognition device in the training stage of the present invention; FIG. 4 is an explanatory diagram showing processing of a face area detection unit and an area division unit; FIG. 2 is a functional configuration diagram of the estimation stage of the facial expression recognition device according to the present invention; It is a functional block diagram of the basic training stage in the program of the present invention. It is a functional block diagram of a basic estimation stage in the program of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

The inventors of the present application, for example, use not only the feature amount of the face exposed area but also the feature amount of the mask wearing area to recognize the facial expression from the face image of the person wearing the mask with high accuracy. I thought it would be better to recognize
At that time, we thought that we should also consider the correlation between the facial expression recognized from the face-exposed area and the facial expression recognized from the mask-wearing area.
As for the exposed face area, facial expressions are likely to appear in the area around the eyes that is not covered by the mask. In particular, facial expressions tend to appear in wrinkles between the eyebrows. On the other hand, the area where the mask is worn is also wrinkled and deformed due to changes in the muscles of the entire face, especially the nose, mouth, and cheeks. The inventors of the present application considered that the facial expression should be recognized by separately estimating the feature amount of the face-exposed area and the feature amount of the mask-wearing area, and considering the correlation between the two.

1 to 3, a technique suitable for recognizing a facial expression from a facial image in which a wearable object is worn on the face will be described below.
4 and 5, techniques suitable for general object recognition applications, not limited to facial expression recognition techniques, will be described.

FIG. 1 is a functional configuration diagram of a training stage of a facial expression recognition apparatus according to the present invention.

The facial expression recognition apparatus 1 can estimate an expression (label) by inputting a person's face image (target image).
According to FIG. 1, the facial expression recognition apparatus 1 includes, as a <training stage>, a teacher image storage unit 2, a face area detection unit 10, an area division unit 11, an exposed face area input intermediate layer (first input intermediate layer) 121 , mask-wearing region input intermediate layer (second input intermediate layer) 122 , intermediate feature value fusion layer 13 , and output layer 14 . These functional components are implemented by executing a program that causes a computer installed in the facial expression recognition apparatus 1 to function. In addition, the flow of processing of these functional components can also be understood as an estimation method of the apparatus.

[Teacher image accumulation unit 2]
The teacher image storage unit 2 stores in advance a large amount of teacher data in which a face image (teacher image) in which a person's face is reflected is associated with a person's expression label (teacher label).
face image (teacher image) <-> facial expression label (teacher label)
In the face image, it is assumed that an object such as a mask is worn on the person's face. Of course, depending on the application, a wearable object such as spectacles, goggles or sunglasses may be worn.
Expression labels may be, for example, positive, negative, and neutral. Of course, four or more facial expressions may be prepared.

The teacher image storage unit 2 outputs each set of face images and expression labels. A face image is input to the face region detection unit 10, and a learning model of each functional component is trained so that an expression label is output from the output layer 14. FIG. Specifically, according to the face image of the teacher data, the face exposed region input intermediate layer 121, the mask wearing region input intermediate layer 122, the intermediate feature amount fusion layer 13, and the output layer 14 are converted from the output layer 14 to the It is trained to output facial expression labels.

FIG. 2 is an explanatory diagram showing the processing of the face area detection section and the area division section.

[Face area detector 10]
The face area detection unit 10 detects a person's face area (for example, a bounding box) from the input face image. The face image input in the <training stage> is a teacher image.
The detected face area is output to the area dividing section 11 .

Specifically, R-CNN (Regions with Convolutional Neural Networks) or SSD (Single Shot Multibox Detector) is used for the face region detection unit 10 .
R-CNN combines square face regions with convolutional neural network features to detect subsets of face regions (region proposal). Next, CNN features are extracted from the region proposal. Then, the bounding box of the region proposal is adjusted by a support vector machine pre-trained using the CNN features.
SSD is a general object detection algorithm using machine learning, which determines rectangular bounding boxes called default boxes. A large number of default boxes with different sizes are superimposed on one image, and a predicted value is calculated for each box. For each default box, we can predict its position, how far it is from the object, and how different it is in size.

[Region dividing unit 11]
The area division unit 11 receives the face area image detected by the face area detection unit 10 and divides it into area images of different area types.
The area dividing unit 11 estimates the area type for each pixel of the input image, and detects the area image based on the boundary line (segmentation) of the area image. Here, the area images of different area types are specifically as follows.
First region type: Face exposed region image Second region type: Mask wearing region image (equipment region image)
Then, the face exposed area image is output to the face exposed area input intermediate layer 121 and the mask wearing area image is output to the mask wearing area input intermediate layer 122 .

The segmenter 11 is a machine learning engine pre-trained with a large dataset of object images and classes.
Specifically, existing technologies such as mask rcnn (registered trademark), YOLACT (registered trademark), and BlendMAS (registered trademark) can be applied. According to mask rcnn, class classification is performed on a pixel-by-pixel basis, and a boundary region based on the class is detected from the entire image. Then, a large number of “object-like regions” are detected from the image. From among them, we narrow down the areas where the "likeness of a human face" is above the threshold and the areas where the "likeness of a mask" is above the threshold, and finally the "face exposed area" and "mask wearing area" are obtained. .
The network structure of mask rcnn is, for example, improved based on Faster R-CNN (see Non-Patent Document 5, for example).

[Face exposed area input intermediate layer 121 and mask wearing area input intermediate layer 122]
The input hidden layer 12 consists of an "input layer" and an "hidden layer", and together with the output layer is based on a general neural network. In particular, the intermediate layer repeatedly trains which part of the image is taken as a feature so that the correct teacher label can be obtained from the output layer.

According to the present invention, the input hidden layer 12 has a model trained to input each region image based on the teacher image and output the teacher label from the output layer 14 .
The input hidden layer 12 functions to extract region features based on the input image. In particular, visualization (heat map) of the (N−1)-th layer at the final stage of the intermediate layer recognizes the feature amount of each area image.

A plurality of input intermediate layers 12 are provided in advance for each region type. According to FIG. 1, the area division unit 11 divides into two area images, and each area image is input to the respective input intermediate layer 12 . Here, the following two are provided.
A face exposed region input hidden layer 121 that trains images of the face region type
A mask-wearing region input hidden layer 122 that trains images of the mask-wearing region type
Of course, the area division unit 11 may divide the image into three area images and have three input intermediate layers.
Then, the face-exposed region input intermediate layer 121 and the mask-wearing region input intermediate layer 122 output intermediate feature amounts to the intermediate feature amount fusion layer 13 .

[Intermediate feature quantity fusion layer 13]
The intermediate feature amount fusion layer 13 fuses the intermediate feature amounts output from the plurality of input intermediate layers 12 and outputs the fused intermediate feature amount to the output layer 14 .
At this time, the intermediate feature value fusion layer 13 derives different weights for each input intermediate layer 12 so that the output layer 14 outputs a teacher label.

The weight of the intermediate feature amount fusion layer 13 is a numerical value of the importance of the feature amount from each input intermediate layer 12 when viewed from the output layer 14 in the neural network.
According to a general neural network, each post-stage neuron trains weights for each of the pre-stage neurons. Weights generally represent the strength of synaptic connections, such that neuron-to-neuron connections change the ease with which information can be transmitted.
On the other hand, the intermediate feature amount fusion layer 13 of the present invention trains weights in units of each input intermediate layer 12 in the previous stage. The weights of the present invention represent the strength of coupling between the face-exposing region input intermediate layer 121 and the mask-wearing region input intermediate layer 122 and the output layer 14 respectively.
Weight for face exposed area input hidden layer 121: β
Weight for mask wearing region input hidden layer 122: 1-β
Here, when recognizing a facial expression, it is assumed that β>(1−β). For facial expression recognition, for example, a weight of β = 0.9 is given to facial expression recognition from the face exposed region, and a weight of 1 - β = 0.1 is given to facial expression estimation from the mask wearing region. assumed to.

On the other hand, in the present invention, the ease of information transfer β between the first input hidden layer and the output layer and the ease of information transfer β−1 between the second input hidden layer and the output layer are output. It is trained from the accuracy of the labels estimated by the layers.

[Output layer 14]
The output layer 14 is a model trained to input a fused intermediate feature value based on a teacher image and output a teacher label. This also has the same function as the output layer in a general neural network.

FIG. 3 is a functional configuration diagram of the estimation stage of the facial expression recognition apparatus according to the present invention.

According to FIG. 3, it basically has the same functional configuration as in FIG.
The facial expression recognition apparatus 1 can estimate a facial expression from a target image in which a person's face is reflected as <estimating stage>. Here, the target image is assumed to be a face image wearing a mask as a wearable object, but of course, a face image not wearing a wearable object may be mixed.
A face area detection unit 10 receives a target image and detects a person's face area image.
The region dividing unit 11 divides a face region image of a person in a target image into a face-exposed region image and a mask-wearing region image as region images of different region types. Then, the face-exposed region image is output to the face-exposed region input intermediate layer 121 and the mask-wearing region image is output to the mask-wearing region input intermediate layer 122 .
The face-exposed region input intermediate layer 121 and the mask-wearing region input intermediate layer 122 each receive an area image and output an intermediate feature amount using a pre-trained model. The intermediate feature amount of the image of the face-exposed area and the intermediate feature amount of the image of the mask-wearing area are separately extracted.
The intermediate feature amount fusion layer 13 fuses intermediate feature amounts output from a plurality of input intermediate layers and outputs a fused intermediate feature amount. As a result, it is possible to fuse the images of the face-exposed region (for example, wrinkles between the eyebrows) and the mask-wearing region (for example, mask wrinkles based on changes in facial muscles) while preserving the tendency of the intermediate feature amount of each image. In particular, by making the weight β of the intermediate feature quantity of the exposed face region heavier than the weight 1−β of the intermediate feature quantity of the mask-wearing region, the result of expression recognition from the exposed face region can be strongly reflected.
The output layer 14 finally receives the fused intermediate feature amount and estimates the expression label using a pre-trained model.

In another embodiment of the present invention, the area dividing unit 11 divides the area image for each mask type, such as a cloth mask, a nonwoven fabric mask, a flat mask, a pleated mask, and a three-dimensional mask. may be In that case, the mask wearing area input intermediate layer 122 is provided according to the mask type. In the training phase, the input hidden layer corresponding to the mask worn on the face is trained according to the teacher image. In the estimation stage, depending on the target image, it is estimated by the input hidden layer corresponding to the mask worn on the face.

FIG. 4 is a functional block diagram of the basic training stage in the program of the present invention.
FIG. 5 is a functional block diagram of the basic estimation stage in the program of the present invention.

4 and 5 are basic functional configuration diagrams that are not limited to applications for recognizing facial expressions from a person's facial image.
The area dividing unit 11 divides the target image into area images of different area types.
A plurality of input intermediate layers 12 are provided for each area type, receive an area image, and output an intermediate feature amount using a pre-trained model.
The intermediate feature amount fusion layer 13 fuses the intermediate feature amounts output from the plurality of input intermediate layers 12 and outputs a fused intermediate feature amount. At this time, it is also preferable that each input intermediate layer 12 is given a different weight.
The output layer 14 inputs the fused intermediate features and estimates labels using pre-trained models.

The functional configurations in FIGS. 4 and 5 can be used for various purposes. For example, if an image is captured by an indoor camera, it may be possible to estimate a label that comprehensively evaluates the entire room by extracting the intermediate feature values separately for the human region image and the furniture region image. unknown. In the case of images taken by an in-vehicle camera, for example, by extracting the intermediate feature values separately for the area image on the road and the area image on the side of the road, a label that comprehensively evaluates the entire traffic can be estimated. maybe you can.

As described in detail above, according to the program, apparatus and method of the present invention, it is possible to analyze each region image of an object separately from the target image and estimate the label by comprehensively evaluating them.
In particular, the present invention can be applied to the use of facial expression recognition, and can estimate the facial expression even if it is a facial image of a person wearing an accessory on the face.

As a result, even in the midst of the corona crisis, "users wearing masks on their faces can estimate their facial expressions without removing the mask." It will be possible to contribute to Goal 3 of the SDGs, “Ensure healthy lives and promote well-being for all at all ages.”

For the various embodiments of the present invention described above, various changes, modifications and omissions within the spirit and scope of the present invention can be easily made by those skilled in the art. The foregoing description is exemplary only and is not intended to be limiting. The invention is to be limited only as limited by the claims and the equivalents thereof.

REFERENCE SIGNS LIST 1 facial expression recognition device 10 facial region detection unit 11 region division unit 12 input intermediate layer 121 face exposed region input intermediate layer, first input intermediate layer 122 mask wearing region input intermediate layer, second input intermediate layer 13 intermediate feature amount Fusion layer 14 Output layer 2 Teacher image storage unit

Claims (8)

In a program that causes a computer to deduce labels from images of interest,
area dividing means for dividing a target image into area images of different area types;
a plurality of input intermediate layers provided for each area type, inputting an area image, and outputting an intermediate feature amount using a pre-trained model;
an intermediate feature value fusion layer that fuses intermediate feature values output from a plurality of input intermediate layers and outputs a fused intermediate feature value;
A program characterized by having a computer function as an output layer for inputting fused intermediate feature values and estimating labels using pre-trained models. 2. The program according to claim 1, wherein the intermediate feature amount fusion layer causes a computer to fuse a plurality of intermediate feature amounts with different weights for each input intermediate layer. The target image is an image in which the face of the person wearing the attachment is reflected,
A label is a person's facial expression,
further comprising face area detection means for inputting a target image and detecting a person's face area image;
3. The program according to claim 1 or 2, wherein the region dividing means causes a computer to function so as to input a face region image of a person and divide it into a face-exposed region image and a wearable object region image as region images of different region types. 4. A program according to claim 3, causing the computer to act like the wearable is a mask, glasses, goggles or sunglasses. Using a teacher image with a teacher label,
The area dividing means divides the teacher image into area images of different area types,
The input hidden layer consists of an input layer and a hidden layer in the neural network, and has a model trained so that each region image based on the teacher image is input and the teacher label is output from the output layer,
The intermediate feature value fusion layer is trained to input the intermediate feature value output from each input intermediate layer and output a teacher label from the output layer, and derives different weights for each input intermediate layer,
5. The output layer according to any one of claims 1 to 4, wherein the computer functions to have a model trained to input fused intermediate features based on a teacher image and output a teacher label. program as described. 5. The area dividing means estimates the type of area for each pixel of the input image and causes the computer to detect the area image based on the boundary line (segmentation) of the area image. The program according to any one of Claims 1 to 3. In an estimation device that estimates a label from a target image,
area dividing means for dividing a target image into area images of different area types;
a plurality of input intermediate layers provided for each area type, inputting an area image, and outputting an intermediate feature amount using a pre-trained model;
an intermediate feature value fusion layer that fuses intermediate feature values output from a plurality of input intermediate layers and outputs a fused intermediate feature value;
An estimating device, comprising: an output layer for inputting a fused intermediate feature amount and estimating a label using a pre-trained model. In a method for estimating a device for estimating a label from a target image,
The device
a first step of dividing the target image into area images of different area types;
a second step of inputting an area image using an input intermediate layer provided for each area type and outputting an intermediate feature amount using a pre-trained model;
a third step of fusing intermediate feature values output from a plurality of input intermediate layers to output a fused intermediate feature value;
and a fourth step of inputting the fused intermediate feature quantity and estimating the label using a pre-trained model.

Images