KR20200106111A - Face landmark detection apparatus and method using gaussian landmark map with regression scheme - Google Patents

Abstract

The present invention can provide an apparatus for detecting a facial feature point and a method thereof. The apparatus includes: a facial area detection unit which receives an input image including a face and detects a facial area to output a facial area map; a feature point prediction unit which detects a feature point from the facial area map according to a pre-designated algorithm and outputs the feature point as a predicted feature point; a Gaussian feature map generation unit for generating a Gaussian feature map having a pixel value according to a Gaussian distribution around the predicted feature point in the facial area map; and a feature point acquisition unit for receiving the facial area map and the Gaussian feature map and detecting the location of the feature point in the facial area map in consideration of the pixel value of the Gaussian feature map according to a pre-learned pattern estimation scheme.

Description

Face feature point detection device and method using Gaussian feature point map and regression technique {FACE LANDMARK DETECTION APPARATUS AND METHOD USING GAUSSIAN LANDMARK MAP WITH REGRESSION SCHEME}

The present invention relates to an apparatus and method for detecting facial feature points, and to an apparatus and method for detecting facial feature points with improved accuracy by using a Gaussian feature point map and a regression technique.

There are various research fields in 2D facial image processing, such as face detection, face recognition, facial expression prediction, and facial feature point detection. Among them, face feature point detection is the most simple expression method for representing facial features. In addition, it is used in various ways as the basis information for various face image processing fields such as pose recognition. In other words, many technologies in the field of facial image processing are performing various application processing on the premise that facial feature points are accurately detected. Therefore, accurate facial feature point detection is very important in the field of facial image processing.

Meanwhile, recently, a learning-based facial feature point detection method such as deep learning has been proposed, and the learning-based facial feature point detection method is configured to detect features by itself using a pre-learned artificial neural network. Currently, artificial neural networks are applied in various ways throughout the latest industries such as voice, video, and data mining, and as one of the representative artificial neural networks in the video field, conball that can extract features that are invariant in size, rotation, and distortion for images. It shows high performance by mainly using a convolutional neural network (CNN). That is, in the case of using CNN, it can exhibit excellent performance in recognizing faces of various shapes for each person such as eyes, nose, and mouth when detecting face specific points. In addition, with the development of CPU and computational technology, the learning-based feature point detection method using artificial neural networks can function effectively in real-time processing.

However, in order to effectively utilize the learning-based facial feature detection method, the artificial neural network must be learned based on a vast amount of data in advance. However, the data set provided for learning is very insufficient to learn all the diversity of facial expressions and poses, and this limits the performance of a learning-based facial feature point detection method using an artificial neural network. In order to solve this lack of training data, a method of increasing the training data set by making some changes such as symmetry, light change, and random patch to a given data set has also been proposed. However, the level of change is not large, so the learning performance is improved. There is a limit that is not large.

Korean Patent Publication No. 10-2015-0127381 (published on November 17, 2015)

An object of the present invention is to provide an apparatus and method for detecting facial feature points capable of accurately detecting facial feature points.

Another object of the present invention is to provide an apparatus and method for detecting facial feature points that can be effectively learned even with a small amount of training data.

In order to achieve the above object, an apparatus for detecting facial feature points according to an embodiment of the present invention includes: a face region detector configured to receive an input image including a face, detect a face region, and output a face region map; A feature point predictor for detecting feature points in the face region map according to a predetermined algorithm and outputting predicted feature points; A Gaussian feature map generating unit generating a Gaussian feature map having pixel values according to a Gaussian distribution centered on the predicted feature point in the face region map; And a feature point acquisition unit receiving the face region map and the Gaussian feature map and detecting a location of the feature point of the face region map in consideration of a pixel value of the Gaussian feature map according to a previously learned pattern estimation method. Includes.

The feature point acquisition unit acquires a correction value for correcting the position of the predicted feature point based on the pixel value of the Gaussian feature map, and corrects the position of the predicted feature point using the obtained correction value, The location of the feature point can be estimated.

The feature point acquisition unit is implemented by a ResNet (Residual Network) including a plurality of convolution boxes each including a plurality of convolution layers and at least one adder that sums the calculation results applied from the previous convolution box, and the plurality of convolutions At least one convolution layer among a plurality of convolution layers included in each of the resolution boxes may have different sizes.

The feature point prediction unit acquires a random forest including a plurality of decision trees generated by randomly sampling the data of the face region map according to a tree algorithm, and uses a regression tree classifier for the feature extracted by the random forest. The predicted feature points may be obtained by predicting the feature points, predicting the facial feature points by repeating a predetermined number of times and correcting the previously predicted facial feature points.

The face region detector may detect the face region using a Histogram of Gradient (HOG) + Support Vector Machine (SVM) method.

In order to achieve the above object, a method for detecting facial feature points according to another embodiment of the present invention includes: receiving an input image including a face, detecting a face region, and outputting a face region map; Detecting a feature point in the face region map according to a predetermined algorithm and outputting a predicted feature point; Generating a Gaussian feature map having pixel values according to a Gaussian distribution centered on the predicted feature point in the face region map; And receiving the face region map and the Gaussian feature map, and detecting a position of the feature point of the face region map in consideration of pixel values of the Gaussian feature map according to a previously learned pattern estimation method. Includes.

Accordingly, the apparatus and method for detecting facial feature points according to an embodiment of the present invention predicts feature points using a tree algorithm for a face region detected through a face detector, and generates a Gaussian feature map based on the predicted feature points. Since the feature bonus is detected by correcting the position of the predicted feature point by applying it as an input of the artificial neural network together with the face region, it is possible to accurately detect the face feature point by effectively learning even with a small amount of training data.

1 shows a schematic structure of an apparatus for detecting facial feature points according to an embodiment of the present invention.
2 and 3 illustrate examples of predicted feature points extracted by the feature point predictor of FIG. 1 using a tree algorithm.
4 shows an example of the structure of the correction value estimating unit of FIG. 1.
5 shows a method of detecting facial feature points according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean units that process at least one function or operation, which is hardware, software, or hardware. And software.

1 shows a schematic structure of an apparatus for detecting facial feature points according to an embodiment of the present invention, and FIGS. 2 and 3 show an example of predicted feature points extracted by the feature point predictor of FIG. 1 using a tree algorithm, and FIG. 1 shows an example of the structure of the correction value estimation unit.

Referring to FIG. 1, the facial feature point detection apparatus according to the present embodiment includes an image acquisition unit 100, a face region detection unit 200, a feature point prediction unit 300, a Gaussian feature map generation unit 400, and a feature map merging unit. 500, a correction value estimating unit 600, and a feature point acquisition unit 700.

The image acquisition unit 100 acquires an input image including a face and transmits it to the face region detection unit 200. Here, the input image is a 2D image captured using a general camera device, and in some cases, it may be a video including a plurality of consecutive 2D images as frames.

The face region detection unit 200 receives an input image for detecting facial feature points from the image acquisition unit 100 and detects a face region from the input image in a predetermined manner. That is, the face region detection unit 200 divides the face region and the background region from the input image, detects only the face region, and outputs a face region map.

Various methods of detecting a face region in an image are known, but here, as an example, it is assumed that the face region is detected using a Histogram of Gradient (HOG) + Support Vector Machine (SVM) method. HOG is a method of detecting the face region by calculating the gradient and orientation of all pixel values within a cell of a certain size, then generating a histogram using these values, and using this as a feature vector of SVM. . However, in this embodiment, the face region detection unit 200 may use a face region detection method other than the HOG+SVM method.

The feature point prediction unit 300 receives the face area map detected by the face area detection unit 200, detects the feature point in a known manner from the applied face area map, and outputs it as a predicted feature point. Various methods of detecting a feature point in a face image are known, but here, as an example, it is assumed that the feature point predictor 300 detects at least one predicted feature point using a tree algorithm.

Face feature point detection using tree algorithm is a recently proposed regression-based face feature point detection method, and it is one of the hand-craft algorithms that detect feature points through the operation of variables and variables for feature point extraction. . The tree algorithm obtains a random forest by randomly sampling data to generate a plurality of decision trees, and extracts features by majority vote by collecting the results of the obtained decision trees of the random forest.

However, at least one predicted feature point detected using the tree algorithm exhibits excellent detection performance in many cases, but there is a problem in that it indicates a very large failure depending on the face image. In particular, a large error may occur depending on the pose of the subject's face included in the image.

Referring to FIG. 2, it can be seen that the predicted feature points detected by the tree algorithm are designated in areas not related to the contours of the face and eyes/nose/mouth. That is, errors can appear large.

Accordingly, the feature point prediction unit 300 applies a regression tree classifier to the feature extracted by the random forest so that it can rapidly converge to the target value. Since the regression tree classifier is used to quickly converge to the target value, the extracted predicted feature points can be refined by repeating a plurality of times in a cascade method. That is, the accuracy of at least one predicted feature point is improved by repeatedly detecting the predicted feature point a plurality of times.

As shown in FIG. 3, it can be seen that when the tree algorithm is refined by repeating the tree algorithm a number of times in a cascade manner using a regression tree classifier, the positional accuracy of the predicted feature points can be greatly improved.

Nevertheless, the facial feature point detection apparatus according to the present embodiment uses the predicted feature point predicted by the feature point predictor 300 as base data for detecting an accurate facial feature point in order to correct an error of the tree algorithm.

The Gaussian feature map generation unit 400 generates a Gaussian feature map having a Gaussian distribution centering on the predicted feature points extracted from the feature point predicting unit 300. The Gaussian feature map generates a Gaussian feature map by assigning pixel values so that surrounding points have different values based on a Gaussian distribution around at least one predicted feature point extracted by the feature point predictor 300. That is, the predicted feature point extracted by the feature point predicting unit 300 generates and outputs a Gaussian feature map to which pixel values are assigned so that the predicted feature point has the largest known value and gradually has a smaller value as the distance from the predicted feature point increases.

The feature map merging unit 500 receives the face region map obtained from the face region detection unit 200 and the Gaussian feature map generated by the Gaussian feature map generation unit 400 and merges the merged map in a predetermined manner to add a correction value. Delivered to the government (600).

When the feature map merging unit 500 merges the Gaussian feature map with the face area map obtained by the face area detection unit 200 and transfers the merged map to the correction value estimating unit 600, the correction value estimating unit 600 It is to be able to refer to the pixel value of the Gaussian feature map as a pixel feature weight when extracting.

Here, the feature map merging unit 500 may create a merge map by merging the face area map and the Gaussian feature map in various ways, but the correction value estimating unit 600 simply combines the data of the face area map and the Gaussian feature map. You can also pass it to. That is, the feature map merging unit 500 is configured to sequentially transfer a face region map composed of two-dimensional vector values and a Gaussian feature map corresponding to the face region map to the correction value estimating unit 600 without a separate processing procedure. It could be.

In this case, the correction value estimating unit 600 may be configured to directly receive a face region map and a Gaussian feature map as inputs from the face region detection unit 200 and the Gaussian feature map generation unit 400. In this case, the feature map merging unit 500 may be omitted.

The correction value estimating unit 600 receives the merge map from the feature map merging unit 500, and extracts a correction value to compensate for the position of each of the at least one predicted feature point from the merge map according to a previously learned pattern estimation method. In particular, in this embodiment, the correction value estimating unit 600 extracts a correction value for the predicted feature point from the face region map by referring to the pixel feature weight included in the Gaussian feature map of the merged map according to the learned pattern estimation method.

The correction value estimating unit 600 may be implemented as a pre-trained artificial neural network, and as an example, as shown in FIG. 4, may be implemented as a Deep Residual Learning for Image Recognition (ResNet) among artificial neural networks. ResNet is an artificial neural network known to be capable of more sophisticated detection performance by maintaining a gradient value as possible even in a deep network structure using a residual learning method.

In FIG. 4, (a) shows a schematic structure and operation of the correction value estimating unit 600 implemented by ResNet, and (b) shows an example of the convolution box (Conv Box) of (a).

ResNet is basically configured based on a convolutional neural network, but a predetermined number of convolutional layers (conv) are grouped into a convolutional box, and each convolutional box (CLB1 ~ CLB4) is Since the adder (AD) has a structure in which a shortcut is connected, it is possible to learn by considering the characteristics extracted by the adjacent convolution box (CLBi-1) before the corresponding convolution box (CLBi) at the time of learning. Even when the depth is very deep, distortion is not generated during pattern estimation, and excellent performance can be exhibited.

Referring to FIG. 4A, the correction value estimating unit 600 according to the present embodiment includes one convolution layer CL1, a plurality of convolution box layers CBL1 to CBL4, and two pulling rays PL1. , PL2) and a fully connected layer (FCL). The convolutional layer CL1 receives a merge map including a face region map and a Gaussian feature map as an input and performs a convolution operation using 64 kernels having a size of 7 X 7. In addition, the first pooling layer PL1 is a max pooling layer, by sliding the feature map obtained as a result of the convolution of the convolution layer CL1 and deriving the maximum value among the values in the window while sliding a 3 X 3 window. Perform. Meanwhile, each of the plurality of convolution box layers (CBL1 to CBL4) includes a plurality of convolution layers (conv) therein, as shown in (b), and each convolution layer (conv) size is configured differently. By configuring the bottleneck architecture, the number of internal parameters can be decreased while increasing the number of feature maps.

In addition, the second pooling layer PL2 is an average pooling layer, and obtains an average value of values within the window while sliding a 7 X 7 window for the feature map output from the final convolution box layer CLB4.

In addition, the fully connected layer FCL classifies and outputs a feature map obtained as an average value in the second pooling layer PL2. The correction value output here may be a distance position information value for moving the predicted feature point.

However, the present invention is not limited thereto, and in some cases, it may be implemented with other artificial neural networks such as CNN.

In addition, the feature point acquisition unit 700 receives the predicted feature point position from the feature point predictor 300 and corrects the applied predicted feature point position according to a correction value applied from the correction value estimating unit 600 to obtain a feature point. That is, the location information of the feature point is acquired.

In the above, for convenience of explanation, the correction value estimating unit 600 and the feature point obtaining unit 700 are shown as separate configurations, but the correction value estimating unit 600 may be included in the feature point obtaining unit 700 and configured. .

Conventionally, when the facial feature point detection device includes an artificial neural network such as the correction value estimating unit 600, the artificial neural network included in the facial feature point detection device is generally trained to extract facial feature points from the face region map. That is, the feature point predictor 300 extracts the predicted feature points, and the artificial neural network directly extracts the facial feature points from the face region map without the need for the Gaussian feature map generator 400 to generate the Gaussian feature map.

However, when the artificial neural network is configured to extract facial feature points directly from the face region map, it is not easy to train the artificial neural network to extract facial feature points because there is insufficient data for learning. In order to overcome this problem, when the position of the predicted feature point extracted by the feature point predictor 300 is input to the artificial neural network together with the face region map, the artificial neural network may be trained to detect the feature point around the predicted feature point. In this way, when an artificial neural network detects a feature point with reference to a predicted feature point, there is an advantage in that the feature point detection performance is improved and the learning time can be reduced. However, as described above, an error may be included in the predicted feature point, and when an error is included in the predicted feature point, the artificial neural network detects the feature point based on the error, and thus the detected feature point also has a limitation in that it may contain an error.

Therefore, in this embodiment, the Gaussian feature map generator 400 generates a Gaussian feature map to which pixel feature weights having a Gaussian distribution are added around the predicted feature points so that the artificial neural network does not place excessive pixel feature weights on the predicted feature points. Provided by Therefore, although the artificial neural network pays more attention to the predicted feature points, it is possible to improve the feature point detection performance by extracting the feature points in a form that disperses the interest even around the predicted feature points without overlying the predicted feature points.

In addition, the artificial neural network does not extract the feature points using the pixel feature weights included in the Gaussian feature map, but is implemented by the correction value estimator 600 to obtain a correction value to correct the predicted feature point position of the face region map. And to improve extraction accuracy. However, in some cases, the feature point acquisition unit 700 may be implemented as an artificial neural network to directly extract the feature points, and the correction value estimation unit 600 may be omitted.

Meanwhile, the apparatus for detecting facial feature points according to the present embodiment may further include a learning unit 800. The learning unit 800 is a configuration for learning the correction value estimating unit 600 implemented as an artificial neural network, and is used in the learning process, and may be excluded when the learning is completed.

In the case of training the correction value estimating unit 600, the facial feature point detection apparatus receives an input image obtained in advance for verification of the feature point (Ground Truth) for learning, and obtains the feature value obtained by the feature point acquisition unit 700. It is acquired and transfers the acquired feature value to the learning unit 800. In addition, the learning unit receives the verification data for the input image, the learning unit 800 determines an error between the location of the feature value acquired by the feature point acquisition unit 700 and the location of the verification data according to a known loss cost function. , The determined error is backpropagated to the correction value estimating unit 600 to train the correction value estimating unit 600. Here, the learning may be performed by updating weights of multiple layers included in the artificial neural network constituting the correction value estimating unit 600.

Referring to the rightmost image in (a) of FIG. 4, the learning unit 800 loses the difference between the feature points in the red verification data, the blue predicted feature points obtained by the tree algorithm, and the yellow previously extracted feature points. By calculating with (Loss) and backpropagating, the correction value estimating unit 600 can be trained. And, as described above, when the feature point acquisition unit 700 is implemented as an artificial neural network and configured to directly extract the feature point, the learning unit 800 backpropagates the error to the feature point acquisition unit 700 and the feature point acquisition unit 700 You can also learn.

5 shows a method of detecting facial feature points according to an embodiment of the present invention.

Referring to FIGS. 1 to 4, the method of detecting facial feature points of FIG. 5 will be described. First, an input image for detecting the feature points is obtained (S10). Here, the input image is a 2D image captured using a general camera device, and may be a video including a plurality of consecutive 2D images as frames.

When the input image is acquired, the face area map is obtained by dividing the face area and the background area from the input image and extracting only the face area (S20). As an example, the face area map may be extracted using a Histogram of Gradient (HOG) + Support Vector Machine (SVM) method.

When the face area map is obtained, a feature point is predicted in a known manner from the applied face area map and output as a predicted feature point (S30). The predicted feature points may be obtained by using a tree algorithm as an example, and in particular, may be obtained by refining by repeating a plurality of times in a cascade manner using a regression tree classifier.

When the predicted feature points are acquired, a Gaussian feature map having pixel values in which neighboring points conform to the Gaussian distribution centered on the acquired predicted feature points is generated (S40). Then, the generated Gaussian feature map is merged with the face area map (S50).

The correction value estimating unit 600 implemented by an artificial neural network learned according to a known pattern estimation technique receives a merge map in which the face region map and the Gaussian feature map are merged and estimates a correction value for correcting the position of the predicted feature point. (S60).

When the correction value is estimated, the position of the predicted feature point is corrected by reflecting the correction value with respect to the position of the acquired predicted feature point, thereby detecting the position of the feature point (S70).

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: image acquisition unit 200: face region detection unit
300: feature point prediction unit 400: Gaussian feature map generator
500: feature map merging unit 600: correction value estimating unit
700: feature point acquisition unit 800: learning unit

Claims (9)

A face region detector configured to receive an input image including a face, detect a face region, and output a face region map;
A feature point predictor for detecting feature points in the face region map according to a predetermined algorithm and outputting predicted feature points;
A Gaussian feature map generating unit generating a Gaussian feature map having pixel values according to a Gaussian distribution centered on the predicted feature point in the face region map; And
A feature point acquisition unit receiving the face region map and the Gaussian feature map and detecting a location of the feature point of the face region map in consideration of a pixel value of the Gaussian feature map according to a previously learned pattern estimation method; Facial feature point detection device comprising a. The method of claim 1, wherein the feature point acquisition unit
By obtaining a correction value for correcting the position of the predicted feature point based on the pixel value of the Gaussian feature map, and correcting the position of the predicted feature point using the obtained correction value, the position of the feature point of the face region map Facial feature point detection device to estimate. The method of claim 2, wherein the feature point acquisition unit
It is implemented as a ResNet (Residual Network) including a plurality of convolutional layers each including a plurality of convolutional boxes including at least one adder that sums the calculation results applied from the previous convolutional box,
At least one convolutional layer among a plurality of convolutional layers included in each of the plurality of convolutional boxes is a facial feature point detection apparatus having different sizes. The method of claim 1, wherein the feature point prediction unit
Obtaining a random forest including a plurality of decision trees generated by randomly sampling the data of the face region map according to a tree algorithm, and predicting facial feature points using a regression tree classifier for features extracted by the random forest. And a facial feature point detection device for obtaining the predicted feature point by repetitively predicting the face feature point by a predetermined number of times to correct the previously predicted face feature point. The method of claim 1, wherein the face area detection unit
A facial feature point detection device that detects a face region by a Histogram of Gradient (HOG) + Support Vector Machine (SVM) method. Receiving an input image including a face, detecting a face region, and outputting a face region map;
Detecting a feature point in the face region map according to a predetermined algorithm and outputting a predicted feature point;
Generating a Gaussian feature map having pixel values according to a Gaussian distribution centered on the predicted feature point in the face region map; And
Receiving the face region map and the Gaussian feature map, and detecting a location of a feature point of the face region map in consideration of a pixel value of the Gaussian feature map according to a previously learned pattern estimation method; Facial feature point detection method comprising a. The method of claim 6, wherein detecting the position of the feature point
Obtaining a correction value for correcting the position of the predicted feature point based on the pixel value of the Gaussian feature map; And
Estimating the position of the feature point of the face area map by correcting the position of the predicted feature point using the obtained correction value; Facial feature point detection method comprising a. The method of claim 6, wherein outputting the predicted feature points
Obtaining a random forest including a plurality of decision trees generated by randomly sampling data of the face region map according to a tree algorithm;
Predicting facial feature points using a regression tree classifier for the features extracted by the random forest: And
Obtaining the predicted feature points by repeating the step of obtaining the random forest and predicting the facial feature points a predetermined number of times to correct the previously predicted facial feature points; Facial feature point detection method comprising a. The method of claim 6, wherein outputting the face area map comprises:
A facial feature point detection method that detects a face region by a Histogram of Gradient (HOG) + Support Vector Machine (SVM) method.

Images