TWI722383B - Pre feature extraction method applied on deep learning - Google Patents

Abstract

A pre feature extraction method applied on deep learning includes: obtaining a first data; extracting a plurality of feature data of a first data by using at least two different feature extraction algorithms; constructing the feature data to a second data; extracting feature data of the second data by using a deep learning algorithms. Therefore, the efficiency and the accuracy of the deep learning algorithms can be enhanced.

Description

Pre-feature extraction method applied to deep learning

The present invention relates to a pre-feature extraction method; more particularly, the present invention relates to a pre-feature extraction method that performs feature extraction in advance before performing deep learning.

Artificial intelligence (AI) is a field of computer science that has received high attention. It is used to solve problems related to human intelligence, such as learning, problem solving, and feature recognition. Machine learning is a way to realize artificial intelligence, that is, to use machine learning as a means to solve artificial intelligence problems. Machine learning has developed into a multi-disciplinary interdisciplinary in the past 30 years, involving probabilistic theory, statistics, approximation theory, convex analysis, computational complexity theory and other disciplines. Machine learning theory is mainly about designing and analyzing some algorithms that allow computers to "learn" automatically. Machine learning algorithm is a kind of algorithm that automatically analyzes and obtains rules from data, and uses the rules to predict unknown data. The process of machine learning generally includes the steps of input data, feature extraction, feature selection, and feature classification. To obtain useful and effective features, experts in various fields need to invest a considerable amount of analysis and research on the data to understand Data characteristics, structure and proper algorithm can only be achieved. Therefore, it can also be called Feature engineering.

Deep Learning is a branch of machine learning. Its purpose is to automatically extract data that is sufficient to represent the characteristics of the data through linear or non-linear transforms in multiple processing layers. Feature. Deep learning has the ability to automatically extract features (feature extraction), and is also regarded as a feature learning (representation learning). It currently has deep development potential and is a feature engineering that can replace experts.

In recent years, deep learning methods have made significant progress in establishing predictive models. This progress comes from breakthroughs in computing power and big data. In traditional shallow learning methods, users must select appropriate indicators to help the system quickly learn features. Deep learning uses a large amount of data so that the system can automatically learn important features. Generally speaking, deep learning is applied to facial expression recognition, and the input data is not processed, and the recognition result is prone to have low accuracy and unstable conditions.

Furthermore, when applying deep learning, various digital data (including: text, sound, video, animation, etc.) are directly input to deep learning for automatic feature extraction, so it is not possible to make a wide range of judgments in response to various situations. On the other hand, when encountering special situations, special methods are required for feature extraction. If the original deep learning is used directly, it is easy to cause unstable recognition results and low accuracy. Moreover, the number of layers of deep learning often increases and the cost increases, but the effect is not good.

Therefore, it is still necessary to improve the current feature extraction methods that apply deep learning.

The present invention uses at least two different feature extraction algorithms to perform pre-feature extraction before performing the deep learning algorithm. As a result, the calculation efficiency and accuracy of the deep learning algorithm in feature extraction can be increased, the number of layers used by the deep learning algorithm can be reduced, and the use cost can be reduced.

In one embodiment of the present invention, a pre-feature extraction method applied to deep learning is disclosed, which includes: obtaining a first data data; using at least two different feature extraction algorithms to perform feature extraction on the first data data at the same time , To obtain the plural feature data of the first data data; gather these feature data into a second data data; and perform feature extraction on the second data data through a deep learning algorithm to obtain the plural features of the second data data data.

In the pre-feature extraction method applied to deep learning in the above embodiment, these feature extraction algorithms are a facial motion coding system feature extraction method, a local binary pattern feature extraction method, or a directional gradient histogram feature extraction method Either of them.

In the pre-feature extraction method applied to deep learning in the above embodiment, the deep learning algorithm can be a deep neural network algorithm or a convolutional neural network algorithm.

In the pre-feature extraction method applied to deep learning in the above embodiment, the first data data can be a static image, a text, a graphic, or a dynamic image.

In the pre-feature extraction method applied to deep learning in the above embodiment, the second data data is a static image, a text, a graphic, or a dynamic image.

In the pre-feature extraction method applied to deep learning in the foregoing embodiment, the number of these feature extraction algorithms is three or more.

In the pre-feature extraction method applied to deep learning in the above embodiment, the second data data may be a multi-dimensional data.

201‧‧‧Image

202‧‧‧Image

211‧‧‧Hidden layer

212‧‧‧Hidden layer

213‧‧‧Hidden layer

214‧‧‧Input layer

215‧‧‧Output layer

h1_0 to h1_29‧‧‧neurons

h2_0 to h2_19‧‧‧ neurons

h3_0 to h3_9‧‧‧neuron

AU1 to AU22‧‧‧Deformation unit

S101, S102, S103, S104‧‧‧Step

301‧‧‧Convolutional layer

302‧‧‧Pooling layer

303‧‧‧Dropout layer

304‧‧‧Flat layer

305‧‧‧Start function Dense Relu

306‧‧‧Start function Dense SoftMax

307‧‧‧Fully connected layer

Fig. 1 is a schematic flow diagram of a pre-feature extraction method applied to deep learning according to an embodiment of the present invention; Fig. 2 is an application of a feature extraction method of a facial motion coding system according to another embodiment of the present invention Schematic diagram; Figure 3 is a schematic diagram showing the application of the local binary mode feature extraction method according to another embodiment of the present invention; Figure 4 is a diagram showing the application of the feature extraction method of directional gradient histogram according to another embodiment of the present invention Schematic diagram; Figure 5 is a schematic diagram showing the architecture of a deep neural network algorithm according to an embodiment of the present invention; Figure 6 is a schematic diagram showing the architecture of a convolutional neural network algorithm according to an embodiment of the present invention; Figure 7 It is a comparison diagram showing the comparison between the pre-feature extraction and the pre-feature extraction before the deep neural network algorithm is performed; and Figure 8 is a comparison diagram showing the comparison of pre-feature extraction and pre-feature extraction before performing the convolutional neural network algorithm.

Please refer to Figure 1. FIG. 1 is a schematic flowchart of a pre-feature extraction method applied to deep learning according to an embodiment of the present invention. A pre-feature extraction method applied to deep learning disclosed in the present invention includes step S101, step S102, step S103, and step S104. Step S101 is to obtain a first data data; Step S102 is to use at least two different feature extraction algorithms to simultaneously perform feature extraction on the first data data to obtain the plural feature data of the first data data; Step S103 is to use this These feature data are collected into a second data data; and step S104 is to perform feature extraction on the second data data through a deep learning algorithm to obtain plural feature data of the second data data.

In the above step S101, the first data data can be a static image, a text, a graphic, or a dynamic image. In the above step S102, at least two different feature extraction algorithms are simultaneously used to perform pre-feature extraction on the first data. More preferably, three or more different feature extraction algorithms are used to perform feature extraction on the first data data in advance. In one embodiment, these feature extraction algorithms can be any two of a facial motion coding system feature extraction method, a local binary pattern feature extraction method, or a directional gradient histogram feature extraction method, or all three are used . In the above step S103, the second data data is composed of plural feature data of the first data data, and the second data data can correspondingly be a static image, a text, a graphic or a dynamic image. In the above step S104 Among them, the deep learning algorithm can be a deep neural network algorithm or a convolutional neural network algorithm.

Please refer to Figure 2. Figure 2 is a schematic diagram showing the application of a facial action coding system feature extraction method according to another embodiment of the present invention; Facial Action Coding System (Facial Action Coding System, FACS) is a famous psychologist P.Ekman and W. Proposed by Professor Friesen. In this system, it is mainly based on the distribution of muscles on the human face, trying to find the changes that can be distinguished by the naked eye, and decomposing the human expression into multiple basic action units (AU) corresponding to the face, and analyzing The action characteristics of these deformation units, the main areas controlled by them, and the expressions related to them are analyzed, and pictures corresponding to the expressions are provided at the same time. In the embodiment in Figure 2, 22 deformation units from AU1 to AU22 are listed. For example, raised eyebrows are AU1, frowns are AU4, and pouts are AU10. The combination of these deformation units corresponding to the face can be used to express any facial expressions of human beings.

An embodiment of the present invention is based on FACS, trying to recognize the rich changes in human expressions, such as raising eyebrows, changes in the corners of the mouth, etc., to improve the previous over-simplified classification methods, and make computer recognition closer to our human perception , More in line with the real world situation. Since most of the changes in facial expressions depend on changes in the size or position of the eye nitrile, eyebrows, and mouth, 12 features are defined in the determination of feature values.

Please refer to Figure 3. FIG. 3 is a schematic diagram showing the application of the local binary mode feature extraction method according to another embodiment of the present invention. Local Binary Pattern (LBP) is an operator used to describe the local texture characteristics of an image; it has significant advantages such as rotation invariance and gray invariance. It was first proposed by T. Ojala, M. Pietikäinen, and D. Harwood in 1994 for local texture feature extraction. In the embodiment in FIG. 3, the image 201 is first divided into 3×3 pixel areas, and the pixel value corresponding to each pixel area is taken out. The pixel value in the central area is used as the threshold, and the value of 8 pixels in the surrounding area is compared. If the pixel value of the surrounding neighboring area is greater than or equal to the threshold, the pixel position of the neighboring area is defined as 1, otherwise it is defined as 0. Next, multiply the thresholded value with the weight of the pixel at the corresponding position and then add the calculated value to be the LBP value. The calculation formula is as follows (1):

Please refer to Figure 4. FIG. 4 is a schematic diagram of the application of the method for extracting features of the directional gradient histogram according to another embodiment of the present invention. Histogram of oriented gradient (HOG) is an image description method for human target detection proposed by French researcher Dalal on CVPR in 2005. It is calculated and counted by the histogram of the gradient direction of the local area of the image. Composition characteristics. As shown in Figure 4, the image 202 is first divided into multiple connected areas, called cell units. Next, obtain the histogram of the gradient or the direction of the edge of each pixel in the cell unit, and finally combine these histograms to form a feature descriptor. Its advantage is that it can maintain quite good invariance to image geometry and optical deformation.

其中，w_i,j為所要得到的參數，η為學習率，c為代價函式，t為第t次參數更新。 Please refer to Figure 5. FIG. 5 is a schematic diagram of the architecture of a deep neural network algorithm according to an embodiment of the present invention. Deep Neural Networks (DNN) are usually referred to as neural networks that contain multiple Hidden Layers 211, 212, and 213. Layer architecture. A deep neural network system includes a multi-layer network composed of an input layer 214, a plurality of hidden layers 211, 212, and 213, and an output layer 215. Only the nodes of each adjacent layer are connected, and the nodes of the same layer and cross-layers are not connected to each other, and each layer can be regarded as a logistic regression model. This layered structure is closer to the structure of the human brain. In addition, DNNs can be trained using backpropagation algorithms. The present invention uses three different feature extraction methods to perform pre-feature extraction, and then combine the extracted feature data into 142-dimensional data as the feature value input of the deep neural network. The hidden layer 211 contains 30 nerves respectively. Element (h1_0 to h1_29), hidden layer 212 contains 20 neurons (h2_0 to h2_19) and hidden layer 213 contains 10 neurons (h3_0 to h3_9), the final prediction image is what kind of expression, you can also classify six The classification value 1 to the classification value 6 for different aspects. The weight update can be solved by the stochastic gradient descent method using the following formula (2):

Among them, w _{i, j} are the parameters to be obtained, η is the learning rate, c is the cost function, and t is the tth parameter update.

The selection of the function of the above formula (2) is related to the type of learning (for example, supervised learning, unsupervised learning, reinforcement learning) and activation function.

Please refer to Figure 6 again. FIG. 6 is a schematic diagram showing the architecture of a convolutional neural network algorithm according to an embodiment of the present invention. Convolutional Neural Network (CNN) is one of the most common deep learning network architectures, because the convolutional layer (Convolutional Neural Network) in the network architecture layer) and pooling layer (Pooling layer) strengthen the pattern recognition (Pattern recognition) and the relationship between adjacent data, so that the application of convolutional neural network to image, sound and other signal types of data types can get very good results . The architecture of the convolutional neural network algorithm of the present invention mainly includes three layers: a convolutional layer 301, a pooling layer 302, and a fully connected layer 307.

As shown in Figure 6, first, three different feature extraction methods are used for pre-feature extraction, and then the feature data obtained by each is combined into 14 x 10 dimensional data data as the feature value input of the convolutional neural network, followed by After the convolution layer 301 is operated by a 3 x 3 convolution kernel (kernel), its corresponding feature map can be extracted. Next, the feature map is compressed in the pooling layer 302. On the one hand, the feature map can be made smaller to simplify network calculation complexity; on the other hand, feature compression can be performed to extract main features. In addition, after convolution and pooling, the Dropout layer 303 is added, the main purpose of which is to avoid overfitting. During the deep learning training process, the Dropout layer 303 will discard a certain proportion of neurons in each batch run according to the probability without calculation, so that each training is like training a different neural network. Next, a flat layer 304 is used to convert the characteristic values into one-dimensional data for the subsequent fully connected layer 307 to use. Then, create a hidden layer in the fully connected layer 307 (please refer to the embodiment in Figure 5), specify the number of neurons to be 128, and use the activation function Dense Relu305 for activation. Finally, it is the output layer (please refer to the embodiment in Figure 5), that is, output six different patterns of classification value 1 to classification value 6. At this time, the startup function Dense SoftMax306 is used to start the output layer.

Please refer to Figure 7. Figure 7 is a comparison diagram showing the comparison of pre-feature extraction and pre-feature extraction before performing the deep neural network algorithm. It should be explained here that the pre-feature extraction in the present invention is referred to as step S102 in the embodiment in FIG. 1. First, the first data data is obtained by using at least two different feature extraction algorithms to obtain the first data data. The complex feature data is then collected into the second data data, and the deep neural network algorithm is to perform feature extraction on the second data data. It should also be mentioned that in an embodiment of the present invention, the complex feature data of the first data data respectively obtained by at least two different feature extraction algorithms are combined into a multi-dimensional second data data for deep learning The characteristic value of the algorithm input. This is of great help to the use of deep learning to extract features of data (such as images) that are more than two-dimensional. The term “no pre-feature extraction” refers to the feature extraction of the first data directly using the deep neural network algorithm. In Figure 7, the expression recognition of an image is taken as an example. It should be noted that the method disclosed in the present invention is not limited to the expression recognition of the image. In Figure 7, when the data corresponding to the image is directly input into the DNN for facial expression recognition without pre-feature extraction, the average accuracy rate obtained through verification is 84.6%, of which the difference between the highest and lowest accuracy rates is 13.76%, and recognition occurs Low rate, large amplitude and unstable conditions. When using the aforementioned three different feature extraction algorithms to perform pre-feature extraction on the data, and then input the obtained feature data into DNN for expression recognition, the average accuracy obtained is 93.91%, which is an increase of 9.31%. The problem of low accuracy is obviously improved, and the difference between the highest and the lowest accuracy is 6.45%, and the error value is improved by nearly half, which also solves the problem of unstable recognition rate.

Figure 8 shows the comparison of the pre-feature extraction and the pre-feature extraction before the convolutional neural network algorithm. Similar results also occur in the convolutional neural network algorithm. When the data corresponding to the image is directly input to CNN for facial expression recognition without pre-feature extraction, the average accuracy rate obtained through verification is 85.86%, of which the difference between the highest and the lowest accuracy rate is 9.93%, and the recognition rate is also low. A large and unstable situation. When using the aforementioned three different feature extraction algorithms to perform pre-feature extraction on the data, and then input the obtained feature data to CNN for facial expression recognition, the average accuracy obtained is 97.91%, which is an increase of 12.05%. The problem of low accuracy is obviously improved, and it is better than the effect achieved by using DNN, and the difference between the highest and lowest accuracy is 2.48%, and the error value is improved by nearly 3/4, which solves the problem of unstable recognition rate. And its effect is still better than using DNN.

Based on the above, the present invention proposes a pre-feature extraction method for deep learning. It first performs various pre-feature extractions for various input data, and then concatenates the acquired feature information into another Data (such as images or images) are classified and identified through deep learning. This method has already performed feature extraction before inputting to deep learning. Therefore, no matter in a specific environment, specific subject or object, specific and useful feature information can be filtered out, and then calculated through deep learning. The method performs feature extraction and selection, and classification and identification, and finally obtains the effect of improving accuracy and stability compared with the original deep learning algorithm only. On the other hand, it can also reduce the burden of deep learning to achieve the advantages of fewer layers, low cost and good results.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone familiar with the art will not depart from the spirit and spirit of the present invention. Within the scope, various changes and modifications can be made. Therefore, the scope of protection of the present invention shall be subject to those defined by the attached patent scope.

S101, S102, S103, S104‧‧‧Step

Claims (5)

A pre-feature extraction method applied to deep learning, comprising: obtaining a first data data from a face image; using at least two different feature extraction algorithms, and simultaneously performing feature extraction on the first data data to obtain The plural feature data of the first data data; the collection of the feature data into a second data data; and the feature extraction of the second data data through a deep learning algorithm to obtain the plural features of the second data data Data; wherein the feature extraction algorithms are any two of a facial motion coding system feature extraction method, a local binary pattern feature extraction method, or a directional gradient histogram feature extraction method; wherein the first data data is A static image, a text, a graphic or a dynamic image. The pre-feature extraction method applied to deep learning as described in item 1 of the scope of patent application, wherein the deep learning algorithm is a deep neural network algorithm or a convolutional neural network algorithm. The pre-feature extraction method applied to deep learning as described in item 1 of the scope of patent application, wherein the second data data is a static image, a text, a graphic or a dynamic image. The pre-feature extraction method applied to deep learning as described in item 1 of the scope of patent application, wherein the number of the feature extraction algorithms is three or more. The pre-feature extraction method applied to deep learning as described in item 1 of the scope of patent application, wherein the second data data is a multi-dimensional data.

Images