CN108921051B - Pedestrian Attribute Recognition Network and Technology Based on Recurrent Neural Network Attention Model - Google Patents

Abstract

The invention provides a pedestrian attribute identification network based on a recurrent neural network attention model and a pedestrian attribute identification technology. The pedestrian attribute identification network comprises a first convolution neural network for extracting the characteristics of a pedestrian whole-body image by using a pedestrian original whole-body image as an input; using the pedestrian whole-body image feature as a first input, using the attention heat map of the attribute group concerned at the previous moment as a second input, and outputting the attention heat map of the attribute group concerned at the current moment and a recurrent neural network passing through the partially highlighted pedestrian feature; and outputting a second convolutional neural network of the attribute prediction probability of the current attention group by using the pedestrian feature subjected to local highlight as an input. The invention utilizes the convolution cyclic neural network attention model to excavate the incidence relation of the spatial positions of the pedestrian attribute regions, more accurately highlights the positions of the regions corresponding to the attributes in the image, and realizes higher pedestrian attribute identification precision.

Description

Pedestrian Attribute Recognition Network and Technology Based on Recurrent Neural Network Attention Model

technical field

The invention belongs to the technical field of neural network and image recognition, and in particular relates to a pedestrian attribute recognition network and technology based on a cyclic neural network attention model.

Background technique

Pedestrian attribute recognition technology can help people automatically complete the task of searching for specific people from massive image and video data. However, due to the low image quality of surveillance video, the small dataset of annotated pedestrian attributes, and the difficulty in obtaining them, it has greatly increased the difficulty of pedestrian attribute recognition from surveillance video images. Existing pedestrian attribute recognition methods based on deep neural networks can be divided into two categories: convolutional neural network (CNN) method and convolutional neural network and recurrent neural network combination method (CNN-RNN). Existing CNN methods such as DeepMAR try to identify each pedestrian attribute from the features of the entire image in isolation. Although this method has achieved certain results, it ignores the spatial locality of pedestrian attributes and the relationship between attributes. It is difficult to obtain higher recognition accuracy. Existing CNN-RNN methods such as JRL methods try to use recurrent neural networks to gradually mine the semantic relationship between pedestrian attributes, such as women wearing skirts, etc., and the recognition accuracy is improved compared with pure CNN methods. However, this method only considers the semantic connections among pedestrian attributes but ignores the spatial locality of attributes. Many attributes of pedestrians focus on a certain area of the image, such as whether to wear glasses and whether to have long hair, which only depends on the visual characteristics of the pedestrian's head area, and other areas are of little use. If the locality of this space is taken into account in the construction process of the pedestrian attribute recognition model, the head area is highlighted when the head attribute is identified, and the interference of background noise is ignored, the pedestrian attribute recognition accuracy can be greatly improved.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems, the present invention provides a pedestrian attribute recognition network based on a recurrent neural network attention model, including:

A first convolutional neural network that uses the original full-body image of the pedestrian as input to extract the feature N(x) of the full-body image of the pedestrian;

Use the pedestrian whole body image feature N(x) as the first input, the attention heatmap A _t-1 (x) of the attribute group concerned at the previous moment as the second input, and output the attention of the attribute group concerned at the current moment A recurrent neural network with heatmap A _t (x) and locally highlighted pedestrian features H _t (x);

Using the locally highlighted pedestrian feature H _t (x) as input, the second convolutional neural network that outputs the attribute prediction probability of the current attention group.

Further, the partially highlighted pedestrian feature H _t (x) uses the attention heatmap A _t-1 (x) of the attribute group concerned at the last moment to act on the pedestrian whole body image feature N (x) Obtained, the calculation formula is as follows:

Further, a batch regularization operation is used on the attribute prediction probability output to combat the identification error caused by the imbalance of positive and negative samples of the attribute.

Further, the pedestrian attribute identification network includes:

For each different attribute group of the original full-body image of the same pedestrian, the memory unit state of the RNN is jointly determined by the locally highlighted pedestrian features of all the predicted attribute groups;

For different prediction moments, the first convolutional neural network shares weights;

The second convolutional neural network shares weights for different prediction moments.

Further, the pedestrian attribute recognition network is trained using a weighted Sigmoid cross-entropy loss function, and the loss function is as follows:

w _f =exp(p _j )

表示模型输出模型对第i个样本预测是否包含第j个属性的概率，y_ij为第i个样本的第j个属性的标签，N为训练样本总数，K为待识别的属性总数。In the above formula, p _j represents the proportion of positive examples of attribute j in the training set, w _j represents the learning weight of positive examples,

Represents the probability that the model output model predicts whether the ith sample contains the jth attribute, y _ij is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified.

The present invention also provides a pedestrian attribute recognition technology based on the cyclic neural network attention model, including:

S1. Obtain a certain number of pedestrian images with attributes to be identified, and mark whether the images have certain or certain attributes, and obtain a data set that can be used to train pedestrian attribute recognition effects; Grouping by spatial neighbor relationship;

S2. Use the combination of the Inception network and the convolutional recurrent neural network to construct a pedestrian attribute recognition network based on the attention model of the convolutional recurrent neural network;

S3. Define the loss function required for training the pedestrian attribute recognition network, and use the training data set obtained in step S1 to train the pedestrian attribute recognition network constructed in step S2;

S4. Use the pedestrian attribute identification network trained in step S3 to identify the attributes in the image of the pedestrian to be identified.

Further, the step S2 includes:

S2-1. Use the Inception network to extract the original full-body image of the pedestrian to obtain the full-body image feature N(x) of the pedestrian;

S2-2. At time i, use the pedestrian whole body image feature N(x) to use the convolutional recurrent neural network to calculate the attention heatmap A _t (x) of the attribute group concerned at the current time, and store the historical information in the volume In the memory unit of the product recurrent neural network;

S2-3. Use the attention heatmap A _t (x) to act on the pedestrian whole body image feature N(x) to obtain the locally highlighted pedestrian feature H _t (x), the calculation formula is as follows:

S2-4. Use the locally highlighted feature H _t (x) to identify attributes of the t-th group of attributes, and output the predicted probability of this group of attributes.

Further, the loss function defined in step S3 is as follows:

w _j =exp(p _j )

表示模型输出模型对第i个样本预测是否包含第j个属性的概率，y_ij为第i个样本的第j个属性的标签，N为训练样本总数，K为待识别的属性总数。In the above formula, p _j represents the proportion of positive examples of attribute j in the training set, w _j represents the learning weight of positive examples,

Indicates the probability that the model output model predicts whether the ith sample contains the jth attribute, y _ij is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified.

Compared with the prior art, the beneficial effects of the present invention are:

The invention utilizes the convolutional cyclic neural network attention model to mine the relationship between the spatial positions of the pedestrian attribute regions, more accurately highlights the positions of the corresponding regions of the attributes in the image, and realizes higher pedestrian attribute recognition accuracy.

Description of drawings

Figure 1 is a structural diagram of a pedestrian attribute recognition network based on a recurrent neural network attention model.

Detailed ways

Example 1

A pedestrian attribute recognition network based on a recurrent neural network attention model, as shown in Figure 1, includes:

A first convolutional neural network that uses the original full-body image of the pedestrian as input to extract the feature N(x) of the full-body image of the pedestrian;

Use the pedestrian whole body image feature N(x) as the first input, the attention heatmap A _t-1 (x) of the attribute group concerned at the previous moment as the second input, and output the attention of the attribute group concerned at the current moment A recurrent neural network with heatmap A _t (x) and locally highlighted pedestrian features H _t (x);

Using the locally highlighted pedestrian feature H _t (x) as input, the second convolutional neural network that outputs the attribute prediction probability of the current attention group.

In the pedestrian attribute recognition network provided in this embodiment, the partially highlighted pedestrian feature H _t (x) uses the attention heatmap A _t-1 (x) of the attribute group concerned at the last moment to act on The calculation formula is as follows:

In the pedestrian attribute identification network provided in this embodiment, a batch regularization operation is used for the attribute prediction probability output to counteract the identification error caused by the imbalance of positive and negative attribute samples.

In the pedestrian attribute identification network provided by this embodiment, it also includes:

For each different attribute group of the original full-body image of the same pedestrian, the memory unit state of the RNN is jointly determined by the locally highlighted pedestrian features of all the predicted attribute groups;

For different prediction moments, the first convolutional neural network shares weights;

The second convolutional neural network shares weights for different prediction moments.

In the pedestrian attribute identification network provided in this embodiment, the pedestrian attribute identification network is trained using a weighted Sigmoid cross-entropy loss function, and the loss function is as follows:

w _j =exp(p _j )

表示模型输出模型对第i个样本预测是否包含第j个属性的概率，y_ij为第i个样本的第j个属性的标签，N为训练样本总数，K为待识别的属性总数。In the above formula, p _j represents the proportion of positive examples of attribute j in the training set, w _j represents the learning weight of positive examples,

Represents the probability that the model output model predicts whether the ith sample contains the jth attribute, y _ij is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified.

Example 2

A pedestrian attribute recognition technology based on a recurrent neural network attention model, including:

S1. Obtain a certain number of pedestrian images with attributes to be identified, and mark whether the images have certain or certain attributes, and obtain a data set that can be used to train the pedestrian attribute recognition effect; then filter all the marked attributes, Then, the attributes obtained by screening are grouped according to semantic and spatial neighbor relationships;

S2. Use the combination of the Inception network and the convolutional recurrent neural network to construct a pedestrian attribute recognition network based on the attention model of the convolutional recurrent neural network, including:

S2-1. Use the Inception network to extract the original full-body image of the pedestrian to obtain the full-body image feature N(x) of the pedestrian;

S2-2. At time i, use the pedestrian whole body image feature N(x) to calculate the attention heatmap A _t (x) of the attribute group concerned at the current time using the convolutional recurrent neural network, and store the historical information in the volume In the memory unit of the product recurrent neural network;

S2-3. Use the attention heatmap A _t (x) to act on the pedestrian whole body image feature N(x) to obtain the locally highlighted pedestrian feature H _t (x), the calculation formula is as follows:

S2-4. Use the locally highlighted feature H _t (x) to identify attributes of the t-th group of attributes, and output the predicted probability of this group of attributes;

S3. Define the loss function required to train the pedestrian attribute recognition network. The loss function is as follows:

w _j =exp(p _j )

表示模型输出模型对第i个样本预测是否包含第j个属性的概率，y_ij为第i个样本的第j个属性的标签，N为训练样本总数，K为待识别的属性总数；In the above formula, p _j represents the proportion of positive examples of attribute j in the training set, w _j represents the learning weight of positive examples,

Indicates the probability of whether the model output model predicts the ith sample to include the jth attribute, y _ij is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified;

Use the training data set obtained in step S1 to train the pedestrian attribute recognition network constructed in step S2; meanwhile, use the test set to test the pedestrian attribute recognition network obtained by training;

S4. Use the pedestrian attribute identification network trained in step S3 to identify attributes in the image of pedestrians to be identified in practical application scenarios.

The following describes the pedestrian attribute identification technology provided by the present invention in detail based on the pedestrian attribute identification RAP data set.

(1) The RAP dataset for pedestrian attribute recognition is used as the dataset for training and testing the effect of pedestrian attribute recognition. The RAP data set is a pedestrian attribute data set compiled by the team of the Institute of Automation, Chinese Academy of Sciences. The data set uses 26 cameras to collect images of pedestrian surveillance videos in shopping malls. Through the analysis of the context information of pedestrian attributes and environmental factors, the final selection is made. 41,585 pedestrian images are added to this dataset; and 72 attributes are annotated for each image, including perspective information, whether there is occlusion, body part information, etc.

(2) The 72 attributes in the RAP dataset are screened, and 51 attributes that need to be used are screened out, and they are divided into 10 groups according to the semantic and spatial neighbor relationship, as shown in Table 1.

Table 1 51 attributes and corresponding groups in the RAP dataset

(3) Construct the pedestrian attribute recognition network shown in Figure 1. The network uses the convolutional recurrent neural network to train the attention models of pedestrian attributes in different groups, and uses the attention model combined with the Inception convolutional neural network for pedestrian attribute recognition. .

(4) Calculate each attribute label in the training set, and the proportion p _j of positive samples to all samples.

(5) Define the loss function required for training the pedestrian attribute recognition network, and bring the p _j calculated in (4) into the calculation, as follows:

w _j =exp(p _j )

(6) Using the stochastic gradient descent algorithm for pedestrian attribute recognition network, the hyperparameters of the training process are set as follows:

The initial learning rate: 0.1, the batch size (Batch Size): 64, the learning rate is reduced to 1/10 of the initial learning rate every 10,000 rounds, and the deep model pre-trained on the Imagenet image classification task is used as the pedestrian attribute recognition model. initial value.

(7) In the actual test scenario, the image to be detected is input into the pedestrian attribute recognition network trained in step (6), and the network outputs 51 prediction probability vectors corresponding to the grouped attributes in step (2) in 10 times. For the probability output corresponding to each attribute, if the probability value is greater than 0.5, it is considered to have the attribute, otherwise it is considered not to have the attribute. The probability output of each attribute is judged in turn, and finally a recognition result of all 51 attributes of pedestrians is output.

Compared with the existing pedestrian attribute identification methods, the pedestrian attribute identification technology based on the cyclic neural network attention model provided by the present invention has higher identification accuracy. The pedestrian attribute recognition technology provided by the present invention obtains a higher evaluation accuracy than the existing CNN method and the CNN-RNN method by evaluating on two current mainstream pedestrian attribute recognition public data sets.

For pedestrian attribute recognition accuracy, mA (mean accuracy, average accuracy) is generally used to measure the pros and cons of attribute recognition algorithms. Due to the uneven distribution of attributes, in order to ensure the rationality of the accuracy calculation results, mA will be calculated separately for each attribute. The accuracy of positive and negative examples is taken as the average of the accuracy of attribute recognition, and then the average of the accuracy of all attributes is integrated to calculate the final mA value of the attribute. The formula for calculating mA is as follows:

Among them, L represents the number of attributes; Pi represents the number of positive examples, TP _i represents the number of correctly predicted positive examples; _Ni represents the number of negative examples, and T _{i represents} the number of correctly predicted negative examples.

Compared with the DeepMAR method proposed in the background art, the mA value of the pedestrian attribute recognition technology proposed by the present invention is increased by 8.76%, and compared with the JRL method by 3.35%. In addition, the pedestrian attribute recognition technology proposed in the present invention is an end-to-end training prediction method, which is very simple, easy to use and efficient in the process of model training and attribute prediction, which is an advantage that the JRL method does not have.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions are made without departing from the spirit and scope of the technical solutions of the present invention, and they should all be included in the scope of the claims of the present invention.

Claims (7)

1. A construction method of a pedestrian attribute identification network based on a recurrent neural network attention model is characterized by comprising the following steps:

a first convolution neural network for extracting the pedestrian whole-body image characteristics N (x) by using the pedestrian original whole-body image as input;

using the pedestrian whole-body image feature N (x) as the first inputIn, attention heat map A of the last time focused attribute group_t-1(x) As a second input, an attention heat map A of the set of attributes of interest at the current time is output_t(x) And pedestrian feature H passing through local highlight_t(x) A recurrent neural network of (a);

using partially highlighted pedestrian features H_t(x) Outputting a second convolutional neural network of the attribute prediction probability of the current attention group as input;

the partially highlighted pedestrian feature H_t(x) Attention heat map A using the set of attributes that were focused on at the previous time_t-1(x) The calculation formula is obtained by acting on the pedestrian whole-body image characteristic N (x) as follows:

H_t(x)＝A_t(x) oN (x) + N (x), and t represents the t-th group.

2. The method of constructing a pedestrian attribute identification network of claim 1 wherein a batch regularization operation is used on the attribute prediction probability outputs.

3. The method for constructing a pedestrian attribute recognition network according to any one of claims 1 to 2, comprising: for each different attribute group of the same original pedestrian whole-body image, the state of a memory unit of the recurrent neural network is determined by the characteristics of the locally highlighted pedestrians of all the predicted attribute groups; sharing the weight value of the first convolution neural network at different prediction moments; the second convolutional neural network shares the weights for different predicted times.

4. The method of claim 3, wherein the pedestrian attribute recognition network is trained using a weighted Sigmoid cross entropy loss function, wherein the loss function is as follows:

W_j＝exp(p_j)

in the above formula, p_jRepresenting the proportion of the positive examples number of the attribute j in the training set, w representing the learning weight of the positive examples,

probability, y, representing whether the model output model includes the jth attribute for the ith sample prediction_ijThe label is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified.

5. A pedestrian attribute identification method based on a recurrent neural network attention model is characterized by comprising the following steps:

s1, acquiring a certain number of pedestrian images with attributes to be identified, marking whether the images have certain attributes or not, and acquiring a data set which can be used for training the identification effect of the attributes of the pedestrians; grouping all the labeled attributes according to semantic and spatial neighbor relations;

s2, constructing a pedestrian attribute identification network based on the recurrent neural network attention model according to any one of claims 1 to 4 by combining an increment network and a convolutional recurrent neural network;

s3, defining a loss function required by training the pedestrian attribute recognition network, and training the pedestrian attribute recognition network constructed in the step S2 by using the training data set obtained in the step S1;

and S4, identifying the attributes in the pedestrian image to be identified by using the pedestrian attribute identification network trained in the step S3.

6. The pedestrian attribute identification method according to claim 5, wherein the step S2 includes:

s2-1, extracting the original pedestrian whole-body image by using an inclusion network to obtain the pedestrian whole-body image characteristic N (x);

s2-2, at the moment i, calculating the relation of the current moment by using the characteristics N (x) of the whole-body image of the pedestrian and a convolution cyclic neural networkAttention heatmap A of property groups of notes_t(x) And storing the historical information in a memory unit of the convolution cyclic neural network;

s2-3, use attention heatmap A_t(x) Acting on the pedestrian whole-body image feature N (x) to obtain the partially highlighted pedestrian feature H_t(x) The calculation formula is as follows:

H_t(x)＝A_t(x) oN (x) + N (x); t represents the t-th group;

s2-4, using the locally highlighted feature H_t(x) And carrying out attribute identification on the t-th group of attributes and outputting the prediction probability of the group of attributes.

7. The pedestrian attribute identification method according to claim 5 or 6, wherein the loss function defined in the step S3 is as follows:

W_j＝exp(p_j)

in the above formula, p_jRepresenting the proportion of the positive examples number of the attribute j in the training set, w representing the learning weight of the positive examples,

probability, y, representing whether the model output model includes the jth attribute for the ith sample prediction_ijThe label is the label of the jth attribute of the ith sample, N is the total number of training samples, and K is the total number of attributes to be identified.

Images