CN114580492B - Cross-domain pedestrian re-recognition method based on mutual learning - Google Patents

Abstract

The invention discloses a cross-domain pedestrian re-identification method based on mutual learning, which comprises two parts of target domain information mining based on mutual learning and training strategies based on mutual learning, wherein the target domain information mining step by utilizing the mutual learning is as follows: (1) Training by using a source domain and a target domain to obtain two pre-training models; (2) Extracting features by using the two models, and excavating neighbor samples of pedestrians in the target domain; (3) generating a pseudo tag by Jaccard distance; the training strategy based on mutual learning comprises the following steps: (1) Each pre-training model selects samples for the peer-to-peer model for training; (2) Defining the isolation of the samples by utilizing KL divergence and selecting the samples; (3) calculating a rank matrix of the selected samples by utilizing KL divergence; (4) Each model is updated through a triplet constructed by the peer-to-peer model; (5) And (5) utilizing the updated two models to carry out target domain information mining, updating the pseudo tag, and carrying out training on the neural network again to finish pedestrian re-identification.

Description

A cross-domain person re-identification method based on mutual learning

Technical Field

The present invention belongs to the field of computer vision technology, and specifically designs a cross-domain pedestrian re-identification method based on mutual learning, which can be used in scenarios such as security monitoring and video analysis.

Background technique

In recent years, as surveillance cameras are spread all over the world, the coverage of surveillance networks is improving comprehensively. The surge in massive data has further exposed the shortcomings of traditional manual security analysis in terms of time efficiency and accuracy. Traditional security analysis systems mainly rely on manual viewing of surveillance content to discover problems. On the one hand, due to the inability to quickly view a large number of videos in a timely manner, the golden emergency time is missed, and on the other hand, the accuracy is not high. Since deep learning was introduced into the field of pedestrian re-identification in 2014, its accuracy has been greatly improved by relying on a large number of labeled data sets and open source network structures. However, when the model trained on the existing labeled data set is applied to a new scene without manual annotation, its performance will drop significantly. This is mainly due to the inconsistent data distribution caused by the illumination, background and camera angle of different scenes. In order to alleviate this problem, the main task of cross-domain pedestrian re-identification is how to transfer the knowledge and feature expression capabilities learned by the model from the labeled data in the source domain to the unlabeled target domain data. Since cross-domain pedestrian re-identification can bring convenience to the practical application of pedestrian re-identification, it has become the main research direction in industry and academia.

At present, the unsupervised cross-domain pedestrian re-identification method is affected by factors such as noisy pseudo-labels, low-quality samples, and differences in the distance distribution of positive and negative samples. The accuracy is still far behind that of supervised pedestrian re-identification, and it is rarely used in actual scenarios and does not fully integrate temporal and spatial information.

Summary of the invention

The technology of the present invention solves the problem: providing a cross-domain pedestrian re-identification method based on mutual learning, which is effectively applied in new scenarios without pedestrian identity annotation, effectively reducing the dependence on annotation in the training model, and can be used in security monitoring, video analysis and other scenarios.

The technical solution of the present invention is as follows: in view of the problem of low quality of pseudo labels generated in previous methods, target domain information mining is performed based on mutual learning, and pseudo labels with higher quality are obtained by combining the feature expression capabilities of two models. First, each model performs neighbor mining according to the k-reciprocal nearest neighbor strategy to obtain a neighbor set, so that two neighbor sets can be obtained for each sample, and then the intersection of the two neighbor sets is taken to obtain a neighbor set with higher confidence. After the two sets are mutually subtracted, there will be positive sample neighbors in the remaining set. For each sample in the set, the feature capabilities of the peer model are used and the k-reciprocal nearest neighbor strategy is used to perform neighbor mining to obtain a neighbor set. By judging the overlap between this neighbor set and the above intersection set, it is judged whether to add this sample to the final neighbor set.

In view of the problem that pseudo-labels are inevitably noisy when used as supervisory information, two models are used for simultaneous training. Each model selects samples for the peer model for training, which effectively alleviates the problem of error accumulation and prevents the problem of model degradation caused by noise. For classification loss, the criterion for sample selection is the size of the cross entropy loss. For triple loss, the criterion for sample selection is isolation. Isolation refers to whether it is possible to find samples with the same identity in this mini-batch with a high probability. If the probability is high, its isolation is low. On the contrary, if the probability is low, its isolation is high. Here, KL divergence is used as an indicator to measure the similarity of identity probability distribution between samples. Samples with high isolation basically have a low similarity with other samples.

A cross-domain pedestrian re-identification method based on mutual learning of the present invention includes: a target domain information mining method based on mutual learning and a training strategy based on mutual learning;

The target domain information mining method based on mutual learning includes the following steps:

(d1) Use the labeled source domain dataset to train to obtain the source domain pre-training model, extract the target domain data features through the source domain pre-training model, use the DBSCAN clustering algorithm to generate pseudo labels for training, and obtain the target domain pre-training model, which are called equivalent models;

(d2) Using the two pre-trained models obtained in step (d1), calculate the neighbor set of each pedestrian in the target domain;

(d3) converting the neighbor set of step (d2) into Jaccard distance, and then generating pseudo labels through DBSCAN clustering algorithm;

The steps of the training strategy based on mutual learning are:

(t1) Each pre-trained model in step (d1) selects the first 80% of the samples with small cross entropy loss for the peer model, and uses this part of the samples to update the parameters of the peer model through the classification loss;

(t2) using the identity probabilities generated by each model updated in step (t1), the KL divergence between each sample is calculated, which represents the identity similarity between samples;

(t3) Based on the KL divergence distance, the isolation of each sample is defined. In step (t1), each model selects the top 80% of samples in isolation for the peer model;

(t4) Using the rank matrix of the sample selected in the KL divergence calculation step (t3), determine the positive and negative sample pairs of the triplet;

(t5) Each model in step (t1) updates parameters using triplet loss based on the triplet constructed by the peer model;

(t6) re-mining the target domain information according to the parameters updated in step (t5), generating updated pseudo labels, and re-training the neural network using the updated pseudo labels;

(t7) After training, the feature model is obtained and pedestrian retrieval is performed.

The step (d1) comprises the following steps:

(d1.1) Use labeled source domain data for training to obtain a source domain pre-training model;

(d1.2) Use the above source domain pre-trained model as the feature model to extract the features of each sample in the target domain, and use the DBSCAN clustering algorithm to generate pseudo labels;

(d1.3) Use the pseudo labels generated in step (d1.2) to perform preliminary training on the target domain dataset to obtain the target domain pre-training model.

The step (d2) comprises the following steps:

(d2.1) Use the two pre-trained models obtained in step (d1) to extract the features of all pedestrians in the target domain dataset and generate two feature matrices. Use the two feature matrices and the k-reciprocal nearest neighbor strategy to find the nearest neighbor samples for each pedestrian.

(d2.2) Perform neighbor mining based on the consistency of the two pre-trained models to obtain a more confident set of neighbor samples and discard the remaining samples after screening;

(d2.3) Each pre-trained model uses the feature expression ability of the peer model to further mine the sample set abandoned in step (d2.2) to obtain valid neighbor samples;

(d2.4) Merge the neighbor samples mined in steps (d2.2) and (d2.3) to obtain the final neighbor set.

The step (d3) comprises the following steps:

(d3.1) According to the neighbor sample set of each sample obtained in step (d2.3), convert it into Jaccard distance to obtain a distance matrix;

(d3.2) According to the distance matrix obtained in step (d3.1), a pseudo label is assigned to each pedestrian sample using the DBSCAN clustering algorithm. In this process, some samples are classified as noise samples;

(d3.3) Use the KNN strategy to assign pseudo labels to the noise samples in step (d3.2);

(d3.4) Combine the pseudo-labels in steps (d3.2) and (d3.3) to obtain the final pseudo-labels of the target domain dataset.

The advantages of the present invention compared with the prior art are:

(1) The present invention focuses on studying the problem of cross-domain pedestrian re-identification and proposes a target domain information mining method based on mutual learning to obtain higher quality supervisory information. At the same time, each model is used to select samples for the peer model to perform network training, thereby alleviating the impact of noise in pseudo-labels on model training. The purpose of the present invention is to use target domain information mining to obtain pseudo-labels as supervisory information when there are no manually labeled pedestrian identities in the target domain, and a model with good feature discrimination ability can also be obtained. It effectively alleviates the problem that the model trained on the manually labeled source domain dataset cannot transfer the feature discrimination ability well to the target domain dataset without manual labeling.

(2) By mutual learning, target domain information is mined. Compared with a single model that mines target domain information to obtain pseudo labels, the present invention can fully utilize the feature expression capabilities of the two models to obtain pseudo labels of higher quality on the basis of obtaining a higher quality neighbor set.

(3) Through mutual learning, network training is performed. Each model selects samples for training for the peer model. The present invention can effectively alleviate the impact of noise in pseudo-labels and alleviate the problem of model degradation caused by noise in supervisory information during training.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the method of the present invention:

FIG2 is a schematic diagram of the pre-training process in the present invention;

FIG3 is a schematic diagram of neighbor sample mining in the present invention;

FIG4 is a schematic diagram of mutual learning based on sample selection in the present invention.

Detailed ways

The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in FIG1 , the present invention provides a cross-domain person re-identification method based on mutual learning, comprising the following steps:

Step 1: Target domain information mining method based on mutual learning

1.1 First, two pre-trained models are needed to generate feature vectors for each pedestrian image. The pre-training process of the two models is shown in Figure 2. Both models use ResNet50 as the backbone network and are pre-trained on the source domain data and the target domain data set respectively. First, an initial model is trained on the source domain data set, and the pseudo-labels of the target domain data are obtained using the initial model. The pseudo-labels are used to preliminarily train an initial model on the target domain. After this step, two pre-trained models _Ms and _Mt can be obtained, where _Ms represents the model trained using the source domain data, and the corresponding _Mt represents the model trained using the target domain data.

1.2 Based on the two pre-trained models, neighbor sample mining is performed, as shown in Figure 3. First, based on the feature expression capabilities of the two models and the k-reciprocal nearest neighbor strategy, two neighbor sample sets are generated for each sample. The formula is as follows:

In formula (1), p represents probe, which is the pedestrian sample for which the nearest neighbor needs to be found; g represents gallery, which is the candidate sample; N(p, k) represents the set of the nearest k neighbors of sample p, where d(p, _gi ) < d(p, _gi+1 ), d(p, g) represents the Euclidean distance between any two pedestrians p and g, and |·| represents the modulus of the set.

Afterwards, neighbor mining is performed based on the consistency of the model. The formula is as follows:

R _co (p, k) = R _s (p, k) ∩ R _t (p, k) (2)

In formula (2), _Rs (p, k) represents the neighbor set obtained by using the feature model trained with source domain data and performing neighbor mining using the k-reciprocal nearest neighbor strategy. Similarly, _Rt (p, k) is the neighbor set obtained by using the feature model trained with target domain data and performing neighbor mining. _Rco (p, k) is the neighbor set obtained by taking the intersection of the two.

After performing a difference operation on R _s (p, k) and R _t (p, k), we get a sample set. We use the feature capability of the peer model to perform neighbor mining again on each sample q in this sample set. By judging the overlap between this neighbor sample and the neighbor set of R _co (p, k), if the overlap is high, we consider that sample q is also a neighbor sample of p. The formula is as follows:

In formula (3), _RT is the result of further mining the remaining neighbor set mined based on the feature model _Mt , and correspondingly, _RS is the result of further mining the remaining neighbor set mined based on the feature model _Ms.

1.3 After obtaining the nearest neighbor set of each sample, convert the nearest neighbor set into Jaccard distance. The formula is as follows:

In the formula, d(p, _gi ) represents the Euclidean distance between two samples, ||·|| ₁ represents the L1 norm, and the vector d _J (p, g _i ) represents the final Jaccard distance.

1.4 After obtaining the Jaccard distance matrix, use the DBSCAN clustering algorithm for clustering. The DBSCAN clustering algorithm will classify some samples as noise samples, and use the KNN algorithm to re-assign pseudo labels to these samples. So far, the pseudo labels of each pedestrian sample are obtained.

Step 2: Network training strategy based on mutual learning

2.1 After obtaining the pseudo-labels, use the pseudo-labels as supervision information for network training. First, formulate the standard for sample selection under mutual learning. Two sample selection criteria are used here, one is to select samples based on the size of the cross entropy loss, and the other is to select samples based on the size of the isolation. Isolation refers to whether it is possible to find samples with the same identity in this mini-batch with a high probability. If a pedestrian can find multiple pedestrian samples with the same identity in the mini-batch, its isolation is low. On the contrary, if the number of pedestrian samples with the same identity that can be found is small or even no pedestrian samples with the same identity can be found, its isolation is high. KL divergence is used here as an indicator to measure the similarity of identity probability distribution between samples. The formula is as follows:

In formula (5), _qi represents the other samples outside the mini-batch except p.

2.2 Sample selection is performed according to the sample selection criteria. This part of the samples is used for classification loss. The sample set obtained according to the size of the cross entropy loss is as follows:

_Dce = argmin _{D′: |D′| = α|D| Lce} (N, D′) (6)

In formula (6), N represents the network, D′ represents the sample set, and L _ce (N, D′) represents the total loss of the current sample set. The sample set obtained according to the isolation size is as follows:

D _iso = arg min _{D′: |D′| = α|D′|} L _iso (N, D′) (7)

In formula (7), _Liso (N, D′) represents the isolated synthesis of the current sample set.

Combining the two sample selection criteria, we get the final sample set. The formula is as follows:

_Dinter ＝{p| _p∈Dce }∩{p| _p∈Diso } (8)

In formula (8), _Dinter is the intersection of the samples originally selected by the two sample selection criteria to obtain a sample set with lower noise that can be used for training.

Define the classifier as C:f→{ _c1 , _c2 , ..., _cn }, and the final classification loss is as follows:

In formula (9), Represents the true probability value of sample i in category j,/> Represents the probability value of the network prediction of sample i in category j,/> is the sample selected based on M _t , and /> is the loss function used to update the model _Ms , corresponding to/> It is the loss function used to update the model M _t . Each model is finally trained and updated using samples selected by the peer model.

2.3 Select samples based on the isolated sample selection criteria and construct triple pairs for the triple loss function. The final triple loss function formula is as follows:

In formula (10), a represents the current sample, which is also the anchor sample in constructing the triplet. d _a,p represents the feature distance between the anchor sample and the positive sample. The positive sample refers to the sample with the same pedestrian identity as the point sample. d _a,n represents the feature distance between the anchor sample and the negative sample. The negative sample refers to the sample with inconsistent pedestrian identity with the anchor sample. The most difficult positive sample pair with the largest similarity distance is selected from all positive sample pairs, that is, the sample pair with the farthest feature distance. The most difficult negative sample pair is selected from all negative sample pairs, that is, the sample pair with the closest feature distance, and finally a triplet is formed. is the sample set selected by Mt according to the isolation principle,/> is the loss function used to update _Ms , corresponding to/> It is the loss function used to update _Mt.

2.4 After obtaining the corresponding loss function, the loss function is used to train the model and update the model parameters. The idea of mutual learning based on sample selection is shown in Figure 4. After training for a period of time, the pseudo label is updated according to the latest model. Figure 4 mainly shows the process of mutual learning training with classification loss and triple loss. Model A and Model B have different feature discrimination capabilities due to different pre-training strategies. Each model selects samples for training for the peer model, so that each model can alleviate the negative impact of training with samples with incorrect pseudo labels.

2.5 After the training is completed, two feature models are obtained. The feature discrimination capabilities of these two feature models are basically the same, and either model can be selected for pedestrian re-identification.

Claims (3)

1. A cross-domain person re-identification method based on mutual learning, characterized by comprising: a target domain information mining method based on mutual learning and a training strategy based on mutual learning; The target domain information mining method based on mutual learning includes the following steps: (d1) Use the labeled source domain dataset to train to obtain the source domain pre-trained model, extract the target domain data features through the source domain pre-trained model, use the DBSCAN clustering algorithm to generate pseudo labels for training, and obtain the target domain pre-trained model, which are called equivalent models to each other; (d2) Using the two pre-trained models obtained in step (d1), calculate the neighbor set of each pedestrian in the target domain; (d3) Convert the neighbor set of step (d2) into Jaccard distance, and then generate pseudo labels through DBSCAN clustering algorithm; The steps of the training strategy based on mutual learning are: (t1) Each pre-trained model in step (d1) selects the top 80% of samples with small cross entropy loss for the peer model, and uses this part of samples to update the parameters of the peer model through classification loss; (t2) Using the identity probabilities generated by each model updated in step (t1), the KL divergence between samples is calculated, which represents the identity similarity between samples; (t3) Define the isolation of each sample based on the KL divergence distance. In step (t1), each model selects the top 80% of samples in isolation for the peer model. (t4) Using the rank matrix of the sample selected in the KL divergence calculation step (t3), determine the positive and negative sample pairs of the triplet; (t5) Each model in step (t1) updates parameters using triplet loss based on the triplet constructed by the peer model; (t6) re-mining the target domain information according to the parameters updated in step (t5), generating updated pseudo labels, and re-training the neural network using the updated pseudo labels; (t7) After training, the feature model is obtained and pedestrian retrieval is performed; The step (d2) comprises the following steps: (d2.1) Use the two pre-trained models obtained in step (d1) to extract the features of all pedestrians in the target domain dataset and generate two feature matrices. Use the two feature matrices and the k-reciprocal nearest neighbor strategy to find the nearest neighbor samples for each pedestrian. (d2.2) Perform neighbor mining based on the consistency of the two pre-trained models, combine the two pre-trained models to jointly select a more reliable set of neighbor samples, and discard the remaining samples after the screening; (d2.3) Each pre-trained model uses the feature expression ability of the peer model to further mine the sample set abandoned in step (d2.2) to obtain valid neighbor samples; (d2.4) Merge the neighbor samples mined in steps (d2.2) and (d2.3) to obtain the final neighbor set. 2. The cross-domain person re-identification method based on mutual learning according to claim 1, characterized in that: the step (d1) comprises the following steps: (d1.1) Use the labeled source domain data for training to obtain the source domain pre-training model; (d1.2) Use the above source domain pre-trained model as the feature model to extract the features of each sample in the target domain, and use the DBSCAN clustering algorithm to generate pseudo labels; (d1.3) Use the pseudo labels generated in step (d1.2) to perform preliminary training on the target domain dataset to obtain the target domain pre-trained model. 3. The cross-domain person re-identification method based on mutual learning according to claim 1, characterized in that: the step (d3) comprises the following steps: (d3.1) According to the set of neighboring samples of each sample obtained in step (d2.3), convert it into Jaccard distance to obtain a distance matrix; (d3.2) Based on the distance matrix obtained in step (d3.1), a pseudo-label is assigned to each pedestrian sample using the DBSCAN clustering algorithm. In this process, some samples are classified as noise samples. (d3.3) Use the KNN strategy to assign pseudo labels to the noise samples in step (d3.2); (d3.4) Combine the pseudo-labels in steps (d3.2) and (d3.3) to obtain the final pseudo-labels of the target domain dataset.