KR101702878B1 - Head pose estimation with accumulated historgram and random forest - Google Patents

Abstract

The present invention relates to a head pose estimation method. More specifically, the head pose estimation method using an accumulated histogram and a random forest comprises: calculating the accumulated histogram of a face image; and classifying the accumulated histogram into a predetermined angle by using the random forest, thereby accurately estimating the rotation direction of the head through a few processes of calculation.

Description

[0001] The present invention relates to a method of estimating head orientation using a cumulative histogram and a random forest,

The present invention relates to a head direction estimating method, more specifically, by calculating a cumulative histogram of a facial image and classifying it into a predetermined angle using a random forest, And a head direction estimation method using a histogram and a random forest.

In the interaction between human and computer, the user's head direction provides much information. Especially, the gaze direction related to the head direction contains not only the user's ID discrimination but also information necessary for the person's intention, estimation of the conversation contents, and distinction of the interested objects. Therefore, head direction recognition is recognized as one of the important preconditions in the implementation of most intelligent interfaces.

The studies for head direction estimation are divided into regional approach and global approach. The regional approach estimates the head direction using facial elements such as eyes, nose, and mouth as shown in Fig. This method starts with the assumption that the position of the facial organs does not largely differ even if the expression changes. Thus, a method of estimating head orientation using structural feature information of face elements with low degrees of freedom is a representative example of a regional approach.

Gee [3] suggests a method of estimating the head direction using the positions of eyes and mouth in the face. Horprasert [4] estimated the head direction more precisely by defining the feature vector using both end points of the eye and nose position. Wang [5] also estimated the head direction based on the positions of both end points of the eye and both end points of the mouth. Fadda [6] used Leonardo da Vinci's human body proportions to estimate the head direction based on the position of the eyes and nose.

In addition to facial features, the relationship between local features is also used.

Maurer [7] estimated the head direction based on the feature vector extracted from the image using Garbor-based Jet without using the structural information of the face element.

Unlike these, we construct the 3D face model and find the head direction using the correspondence relation between the feature point and the model. Kong [8] constructed a 3D facial model and estimated the head direction using the correspondence between the model's endpoints, both ends of the eye, both ends of the mouth, and the model.

Local approaches are based on the premise that face feature vectors must be detected accurately. However, it is difficult to detect accurate feature vectors due to various factors such as individual differences of faces and shading of faces due to illumination, and there is a problem that the results are sensitively changed according to the detected feature vectors.

The global approach is to use the entire facial image for head orientation estimation. The method of estimating the head direction compared with the template image is the most representative method.

Niyogi [9] developed a system to estimate the head direction using TSVQ (Tree Structured Vector Quantization) and low-resolution facial image template. Lanitis [10] used the Flexible Model based on contour lines as a template to obtain the position information of facial elements and to estimate the direction of the head. Sumi [11] estimated the approximate direction of the face by using the individual template of facial elements and matching the template with the recorded position information of them. Template-based methods can be affected by personal differences in the body, background, lighting, and so on, which can limit the accuracy of acquired features.

In addition, there is a method of estimating the head direction using a multi-detector. Huang [12] estimated the head direction using a boosting method that combines several classifiers to form a tree structure. There is also a method of estimating the head direction using a nonlinear regression method. Gourier [13] estimates the head orientation using a linear auto-associative memory, which learns faces by using color-difference-based features, obtains imagettes by normalizing them, and learns them with each pose using Widrow-Hoff learning method.

The local approach can more accurately estimate the direction than the template-based method if the feature vector extraction is well done.

[Prior Art Literature]

[Non-Patent Document]

[1] http://www.jdl.ac.cn/peal/JDL-PEAL-Release.htm

[2] L. Breiman, "Random Forests," Machine Learning, vol. 45, no. 1, pp.53,23, 2001.

[3] A. Gee and R. Cipolla, "Non-intrusive Gaze Tracking of Human Computer Interaction," Cambridge University, 1995.

[4] T. Horprasert, Y. Yacoob and L.S. Davis, " Computing 3-D Head Orientation from a Monocular Image Secuqence, " Proc. 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp. 242-247, 1996.

[5] J.G. Wang and E. Sung, " EM Enhancement of 3D Head Pose Estimated by Point at Infinity, " Image and Vision Computing, vol. 25, no. 12, pp. 1864-1874, 2007.

[6] G. Fadda, G. L. Marcialis, F. Roli, L. Ghiani, "Exploiting the Golden Ratio on Head-Pose Estimation." In: Image Analysis and Processing? ICIAP 2013, Springer Berlin Heidelberg, pp. 280-289, 2013.

[7] T. Maurer and C. von der Malsburg, "Tracking and Learning Graphs and Pose on Image Sequences of Faces," Proc. 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp. 176-181, 1996.

[8] S.G. Kong and Ralph Oyini Mbouna, " Head Pose Estimation From a 2D Face Image Using 3D Face Morphing With Depth Parameters, "IEEE Trans. On Image Processing, Vol.24, No.6, pp. 1801-1808,

[9] S. Niyogi and W. Freeman, "Example-Based Head Tracking", Proc. 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp. 374-377, 1996.

[10] A. Lanitis, C.J. Taylor, T.F. Cootes and T. Ahmed, " Automatic Interaction of Human Faces and Hand Gestures Using Flexible Models ", Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp. 98-103, 1995.

[11] Y. Sumi and Y. Ohta, "Detection of face orientation and facial components using distributed appearance modeling", Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp. 254-259, 1995.

[12] C. Huang, H. Ai, Y. Li and S. Lao, "High-performance rotation invariant multiview face detection," IEEE Trans. on Pattern Ana-lysis and Machine Intelligence, Vol.29, No. 4, pp. 671-686, 2007.

[13] N. Gourier, J. Maisonnasse, D. Hall and J.L. Crowley, " Head pose estimation on low resolution images, " Lecture Notes in Computer Science 4122, pp. 270-280, 2007.

An object of the present invention is to provide a head direction estimation method capable of improving the accuracy of estimation by effectively reflecting regional characteristics of a face image with a small amount of calculation in estimating head direction in an image.

According to an aspect of the present invention, there is provided a method for designing a classifier, the method comprising: designing a classifier capable of classifying a face direction using facial image learning data; A step of pre-processing and normalizing an input image to be classified to obtain a face image; Converting the facial image into a binary edge image (hereinafter referred to as a 'target edge image'), and calculating a difference image between the average front edge image and the target edge image; And generating an accumulated histogram of the difference image; And inputting the cumulative histogram to the classifier to estimate a y-axis rotation angle and an x-axis rotation angle of the facial image.

In a preferred embodiment, the preprocessing is a process of extracting a head region by removing noise and background from the input image.

In a preferred embodiment, the removal of the noise is performed by Gaussian filtering.

In a preferred embodiment, the normalization is a process of normalizing the size of the head area by enlarging or reducing the size of the head area to a specific size.

In a preferred embodiment, the subject edge image is obtained through Canny Edge Detection.

In a preferred embodiment, the classifier is a decision tree learned in a random forest method.

In a preferred embodiment, the cumulative histogram includes an x-axis cumulative histogram that accumulates the edge distribution of the difference image on the x-axis and a y-cumulative histogram that accumulates on the y-axis.

In a preferred embodiment, the classifier receives the x-axis cumulative histogram and the y-axis cumulative histogram and estimates the y-axis rotation angle and the x-axis rotation angle of the face image, respectively.

The present invention further provides a computer program stored in a computer-readable medium for performing the head-direction estimating method in combination with a computer.

The present invention further provides a computer including a memory, a central processing unit, and an input / output device, wherein the computer program is stored to perform a head-direction estimation method.

The present invention further provides a server computer including a memory, a central processing unit, an input / output device, and a communication device, the memory storing the computer program, and transmitting the computer program to a client computer via a communication network.

The present invention has the following excellent effects.

According to the head direction estimation method of the present invention, the cumulative histogram of the face image is used as the face feature, so that it is possible to accurately estimate the head rotation angle by reflecting local characteristics even with a small amount of calculation.

Further, according to the head direction estimation method of the present invention, there is an advantage that reliability of estimation can be improved by using a random forest algorithm.

1 is a flowchart of a head direction estimating method according to an embodiment of the present invention,
2 illustrates an example of facial image learning data used for classifier learning in the head direction estimation method according to an embodiment of the present invention,
FIG. 3 illustrates an example of a decision tree learned by the random forest method in the head direction estimation method according to an embodiment of the present invention,
4 is a view for explaining a process of extracting a normalized face image from an input image in a head direction estimation method according to an embodiment of the present invention;
5 is a diagram for explaining a process of calculating an accumulated histogram in a head direction estimation method according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating the head direction estimation result estimated by the head direction estimation method according to an embodiment of the present invention.

Although the terms used in the present invention have been selected as general terms that are widely used at present, there are some terms selected arbitrarily by the applicant in a specific case. In this case, the meaning described or used in the detailed description part of the invention The meaning must be grasped.

Hereinafter, the technical structure of the present invention will be described in detail with reference to preferred embodiments shown in the accompanying drawings.

However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Like reference numerals designate like elements throughout the specification.

The head direction estimation method according to an embodiment of the present invention estimates how much the head of a person included in the input image rotates about the x axis and the y axis in the image, It is a technology that can be applied.

Further, the head-direction estimating method according to an embodiment of the present invention is performed by a computer in which a computer program for substantially head-direction estimating is stored.

In addition, the computer includes a memory, a central processing unit, and an input / output device for storing the computer program in a broad concept including a smart device such as an embedded system such as a security camera, a smart phone, and a tablet PC as well as a general personal computer Any device capable of performing image processing is possible.

In addition, the computer program may be stored in a recording medium separately from the computer, and the recording medium may be designed and configured specifically for the present invention, or may be publicly known and used by a person having ordinary skill in the computer software field Optical recording media such as CD and DVD, magneto-optical recording media which can also serve as magnetic and optical recording media, ROM, RAM, flash memory, and flash memory. Or the like, or a hardware device specially configured to store and execute program instructions by itself or in combination.

In addition, the computer program may be a program consisting of program commands, local data files, local data structures, etc., alone or in combination, and may be executed by a computer using an interpreter or the like as well as machine code Lt; RTI ID = 0.0 > language code. ≪ / RTI >

The computer program may be stored in a server computer and transmitted to a client computer through a communication network and downloaded.

In addition, the server computer may include a memory capable of storing the computer program, a central processing unit, an input / output device, and a communication device.

Hereinafter, a head direction estimation method according to an embodiment of the present invention will be described in detail with reference to FIG.

Referring to FIG. 2, in the head direction estimation method according to an embodiment of the present invention, a classifier capable of classifying a predetermined input vector using face image learning data is designed (S1000).

In the present invention, a decision tree learned by a random forest method is used as the classifier.

Random Forest [2] is a decision tree meta-learning method in which a traditional decision tree technique is extended to a number of tree structures by an algorithm proposed by L. Breiman in 2001. [ It is easier to design the classification model because the tree that constitutes the random forest is generated only through the growth process, omitting the pruning process which is difficult to handle in the general decision tree. This is because, based on the 'large number of rules', it is possible to achieve an accuracy that exceeds the performance of a single decision tree with pruning by combining the results of sufficiently many different random decision trees.

As the randomness of the random forest increases, the relevance of each decision tree decreases, which directly contributes to the improvement of the accuracy of the algorithm. Elements that can give randomness are learning data selection of each tree and optimization process at each node. The random forest learning process considering this is as follows.

First, a bootstrap is formed as many as the number of decision trees to be generated from the learning set. Bootstrap is a subset that is generated by repeating arbitrary restoration extraction from the learning set and having the same size as the original learning set. If we create each tree using bootstrapping, there will be a difference in the learning image. Therefore, the branching function of the first node of each tree changes, and various types of trees can be obtained. That is, bootstrap is a device that gives randomness to selection of a learning image.

After the bootstrap generation, a crystal tree is grown from each bootstrap. Unlike a general decision tree, a tree constituting a random forest generates a branch function considering only all the feature values of the learning image or the feature values of the learning image at random, do. Thus, even if the same learning image is given, it can be grown into a tree having a different shape. That is, the method of assigning randomness to the number of feature values or dimension is used as a device for giving randomness to node optimization. In this case, the precision of each individual random decision tree may be lowered, but the accuracy and stability of the random forest to perform the prediction by combining them are increased as described above.

The criteria for stopping growth of each decision tree are as follows. First, if the number of learning data arriving at a given node is smaller than a certain value, it may lose generality by handling too little data to re-partition the data at that node, thus stopping the growth and creating leaf nodes. Second, when the depth value of the decision tree reaches a certain value, the tree is completely grown up to the corresponding depth, and as the depth value becomes larger, the total calculation amount increases exponentially. Therefore, . Third, if the amount of information obtained by dividing the data is less than a certain value, the influence of the node on the accuracy of the tree is very small, so the growth is stopped and a leaf node is generated.

FIG. 3 shows an example of generating a decision tree (T ₁ , T ₂ ) using the random forest. FIG. 3 shows an example in which a decision tree is generated using an image feature existing in an image as an arbitrary dimension value.

Random forests have similarities with other decision trees because they have less randomness as the number of dimensions to consider when optimizing nodes increases. Most of them are generated as similar trees. In this case, they have similar performance to single decision trees. Therefore, if the number of dimensions to be considered is small and the randomness is increased, the correlation with other decision trees is reduced. Therefore, various types of trees are generated and combined to obtain accurate results than a single decision tree.

In the present invention, accumulated histograms (H1 and H2) are used as image features. A detailed description of the cumulative histogram will be given in Figs. 4 and 5.

FIG. 2 shows the face image learning data used in the learning of the decision tree. In the present invention, CAS-PEAL Dataset [1] is used. This data is composed of images including various variable elements such as pose change, facial expression change, wearing of ornaments, illumination condition change for 595 male and 445 female 1040 persons.

The database for head direction recognition is composed of 21 (7 * 3) different image groups for each individual. Of these, 101 had rotations with respect to the vertical axis (-67 °, -45 °, -22 °, 0 °, 22 °, 45 ° and 67 °), 939 (45 °, -30 °, -15 °, 0 ° °, 15 °, 30 °, 45 °) and the rotation about the horizontal axis is the same (-30 °, 0 °, 30 °). In the present invention, only the pose images of the latter 939 persons were used.

That is, the classifier can classify the angle of rotation of the face with respect to the vertical axis (y axis) into seven angles of 45 °, -30 °, -15 °, 0 °, 15 °, 30 ° and 45 °, (x-axis) can be classified into three angles of -30 °, 0 °, and 30 °.

Next, the input image which is the target image to be classified is inputted (S2000).

Referring to FIG. 4, the input image 100 includes an image including a background 120 as well as a human head region 110.

Next, the preprocessing and normalization of the input image 100 is performed (S3000).

The preprocessing process includes removing the background 120 from the input image 100, extracting the face image 111 including the head region 110, and removing unnecessary noise.

In addition, the extraction of the facial image 111 can use a known head region extraction method [4], but there is no particular limitation.

Also, in the present invention, noise generated under different illumination conditions is removed by using a Gaussian filter.

In addition, the normalization is a process of converting the face image 111 into a size and a brightness having a comparable standard type and obtaining normalized face images 200 by normalization.

In the present invention, the face image 111 is normalized to a 160x90 size image, but can be changed.

Next, the normalized facial image 200 is converted into a binary edge image (S4000).

5, the target edge image 210 is an image obtained by binarizing an edge portion of the face image 110. In the present invention, the edge image 210 is acquired through Canny Edge Detection.

Next, a difference image between the target edge image 210 and the average front edge image 300 is calculated (S5000).

In addition, the average front edge image 300 preprocesses and normalizes the face image learning data shown in FIG. 2, and averages the images having the rotation amount of '0' with respect to the x axis or the y axis, It is the image which is converted into the binary edge image by giving the threshold value.

The difference image between the target edge image 210 and the average front edge image 300 is the edge of the portion of the average front edge image 300 that is different from the face element (mouth, eye, nose, eyebrow, So that the change information according to the rotation of the face can be obtained when the cumulative histogram to be described below is calculated.

Next, an accumulated histogram of the difference image is generated (S6000).

5, the cumulative histogram 400 may include a y-axis cumulative histogram 410 in which the edges of the difference image are accumulated in the y-axis and an x-axis cumulative histogram 420 in the x-axis Which is used to estimate the rotation angle in the y-axis direction and the rotation angle in the x-axis direction, respectively.

In addition, the accumulated histogram may be subjected to histogram smoothing and normalization.

That is, estimating the rotation angle of the face using the cumulative histograms 410 and 420 is the most important part of the present invention, and extracts the positions of the conventional facial features, i.e., eyes, nose, and mouth, When the method fails to extract the exact position of each organ, the head direction estimation accuracy is very low. On the other hand, since the present invention uses the edge information and the accumulated histogram, the accuracy of the estimation can be improved, And subtraction. Therefore, it is very suitable for real-time detection because the processing speed can be greatly improved.

In addition, the decision tree shown in FIG. 3 is also designed by using the generated cumulative histogram as the learning data after generating the cumulative histogram through the preprocessing, normalization, and edge image conversion process of the face image learning data of FIG.

Next, the cumulative histograms 410 and 420 are input to the classifier as feature vectors (S7000). The classifier classifies angles of the cumulative histograms 410 and 420 rotated in the x and y axes, (S8000), and the process ends.

FIG. 6 shows the head direction estimation result estimated by the head orientation estimation method according to an embodiment of the present invention. The rotation of the CAS-PEAL Dataset shown in FIG. 2 with respect to the vertical axis is (45 °, -30 °, Of the 939 persons whose rotation is about -30 degrees, -15 degrees, 0 degrees, 15 degrees, 30 degrees, 45 degrees) and the rotation about the horizontal axis is (-30 degrees, 0 degrees, 30 degrees) And 239 were used as estimation subjects.

Referring to the estimation result shown in FIG. 6, when the head is directed to the front, the direction can be recognized almost completely and the recognition rate is lowered when the head is rotated leftward or rightward or upward or downward. However, And the recognition rate was high.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the present invention. Various changes and modifications will be possible.

100: Input image 110: Head area
111: Facial image 120: Background
200: Normalized facial image 210: Target edge image
300: Average front edge image 400: Cumulative histogram

Claims (11)

Designing a classifier capable of classifying a face direction using face image learning data;
A step of pre-processing and normalizing an input image to be classified to obtain a face image;
Converting the facial image into a binary edge image (hereinafter referred to as a 'target edge image'), and calculating a difference image between the average front edge image and the target edge image; And
Generating an accumulated histogram of the difference image;
And inputting the cumulative histogram to the classifier to estimate a y-axis rotation angle and an x-axis rotation angle of the face image.
The method according to claim 1,
Wherein the preprocessing is a process of extracting a head region by removing noise and background from the input image.
3. The method of claim 2,
Wherein the removal of the noise is performed by Gaussian filtering.
3. The method of claim 2,
Wherein the normalization is a process of normalizing the size of the head region by expanding or reducing the size of the head region to a specific size.
The method according to claim 1,
Wherein the subject edge image is obtained through Canny Edge Detection.
The method according to claim 1,
Wherein the classifier is a decision tree learned by a random forest method.
7. The method according to any one of claims 1 to 6,
Wherein the cumulative histogram includes an x-axis cumulative histogram that accumulates the edge distribution of the difference image on the x-axis and a y-cumulative histogram that is accumulated on the y-axis.
8. The method of claim 7,
Wherein the classifier receives the x-axis cumulative histogram and the y-axis cumulative histogram and estimates the y-axis rotation angle and the x-axis rotation angle of the face image, respectively.
A computer program stored in a computer-readable medium for performing the head-direction estimating method of claim 7 in combination with a computer.
A computer comprising a memory, a central processing unit, and an input / output device, wherein the computer program of claim 9 is stored to perform a head-direction estimation method.
A server computer comprising a memory, a central processing unit, an input / output device, and a communication device, wherein the computer program of claim 9 is stored in the memory, and the computer program can be transferred to the client computer through a communication network.

Images