CN117935172A - Visible light infrared pedestrian re-identification method and system based on spectral information filtering - Google Patents

Abstract

The present invention discloses a visible light infrared pedestrian re-identification method and system based on spectral information filtering, the method comprising the following steps: (1) obtaining raw data, dividing it into a training set, a validation set and a test set and performing preprocessing; (2) randomly forming a cross-modal image pair with the obtained batch training samples; (3) building a three-branch pedestrian re-identification network based on PyTorch and setting training parameters; (4) dividing the training period into two stages, V-T and V-I. When in the V-T stage, calculating the semantic consistency loss to update the network weights, taking the transition mode as a filtering condition, and retaining the spectral information most relevant to the infrared mode from the visible light mode; (5) calculating the cascade aggregation loss when in the V-I stage, updating the network weights, directly realizing modal alignment between the visible light and infrared modes, and extracting modal shared representations; using the validation set to verify the accuracy of the algorithm, and saving the network weights with the optimal accuracy.

Description

A visible light infrared pedestrian re-identification method and system based on spectral information filtering

Technical Field

The present invention relates to the technical field of traffic environment, and in particular to a visible light infrared pedestrian re-identification method and system based on spectral information filtering.

Background technique

Person re-identification aims to accurately identify and match the same pedestrian at different locations or time points by analyzing the visual features of pedestrians in different scenes or cameras. In recent years, this technology has received increasing attention due to its wide application in the fields of intelligent security and smart transportation. With the continuous development of deep learning and neural network architecture, pedestrian re-identification has achieved remarkable results. However, the design of most current methods mainly focuses on single-modal pedestrian re-identification (visible light), ignoring that visible light images are easily affected by lighting conditions during the imaging process. In poor lighting conditions or night environments, the captured visible light images lack sufficient visual cues to accurately identify the identity. Therefore, in order to compensate for this limitation, recent research has gradually shifted its focus to visible light infrared pedestrian re-identification, making full use of the advantages of infrared images in low-light environments to provide a more reliable solution for identity recognition.

Visible-infrared person re-identification not only has to deal with the inherent challenges of the person re-identification task (such as viewpoint change, posture change, illumination difference, etc.), but also has to overcome the significant differences between the two different sensor modalities. This modality difference stems from the different physical principles used in the acquisition process of visible light and infrared images. Visible light images are directly affected by natural lighting conditions and capture the surface color and texture of objects, while infrared images are based on the thermal radiation of the target object and focus more on the temperature distribution of the object. These two different physical properties lead to obvious differences in pedestrian appearance, texture, brightness, and thermal distribution. Existing methods can generally be divided into two categories: (1) modality-sharing feature learning, which aims to mine the common representation of cross-modal pedestrian images through metric learning or modality-specific information disentanglement; (2) modality compensation learning, which aims to generate missing modality attributes at the image or feature level and alleviate the modality difference through attribute completion. However, these two types of methods attempt to directly deal with the huge modality difference, but ignore the fact that the large modality gap is not conducive to exploring the spectral correspondence between visible light and infrared images, and it is difficult to learn sufficient discriminative semantics.

Summary of the invention

Purpose of the invention: The purpose of the present invention is to provide a visible light infrared pedestrian re-identification method and system based on spectral information filtering to improve and optimize the existing visible light infrared pedestrian re-identification algorithm, capture the potential spectral correspondence between modalities, and solve the problem of low feature matching accuracy.

Technical solution: The visible light infrared pedestrian re-identification method based on spectral information filtering described in the present invention comprises the following steps:

(1) Obtain the original data, divide it into training set, validation set and test set, and perform preprocessing;

(2) Randomly group the batch training samples processed in step (1) into cross-modal image pairs;

(3) Build a three-branch person re-identification network based on PyTorch and set training parameters. Use the original training samples and synthesized transition images as network input to extract pedestrian representations.

(4) The training period is divided into two stages: V-T and V-I. When in the V-T stage, the semantic consistency loss is calculated to update the network weights, and the transition mode is used as a filtering condition to retain the spectral information most relevant to the infrared mode from the visible light mode;

(5) When in the V-I stage, the cascade aggregation loss is calculated, the network weights are updated, modal alignment is directly achieved between the visible light and infrared modalities, and the modal shared representation is extracted; during the training process, the verification set is used to verify the accuracy of the algorithm and save the network weights with the optimal accuracy.

Furthermore, in step (1), the original dataset is the publicly available SYSU-MM01 or RegDB; the preprocessing includes: the training samples are cropped and scaled to 288x144 pixels, and the diversity of the samples is increased by horizontal flipping and random erasing, and finally the channel mean and standard deviation calculated from the ImageNet dataset are used for standardization.

Furthermore, the step (2) is specifically as follows: first, randomly select one of the red, green, and blue color channels of the visible light image and expand it into three channels; then, divide the processed visible light image and the original infrared image horizontally into multiple parts, and each part retains one of the two modes with the same probability; finally, splice them along the height dimension of the image; each image pair generates a transition image; the formula is as follows:

Suppose we take out the visible light image from the image pair Randomly select one of the three channels of red, green and blue and expand it into three channels, then transform the visible light image and the original infrared image/> The horizontal division is part; then:

;

in, 、/> 、/> Respectively represent the red, green and blue channels in the visible light image, /> and/> For random selection and expansion operations;

;

in, and/> An array of horizontal stripes of the image. Indicates a random selection operation on array elements to obtain a new array; /> It is the stitching operation of the image height dimension.

Furthermore, the step (3) is specifically as follows: select the ResNet50 model pre-trained on ImageNet as the feature extractor, and then fine-tune the network on the SYSU-MM01 or RegDB dataset; wherein, ResNet50 has a total of five stages, the first stage is replicated three times as the input ports of the visible light, transition and infrared modes respectively, and the last four stages are modality-shared to form a three-branch pedestrian re-identification network; the visible light and infrared images read in step (1) and the transition images synthesized in step (2) are input into the pedestrian re-identification network to extract pedestrian features.

Furthermore, the step (4) is specifically as follows: using the pedestrian features extracted in step (3) and the identity tag information read in step (1), a joint training loss consisting of a basic loss and a semantic consistency loss is calculated, and the pedestrian re-identification network is updated by gradient back propagation; the semantic consistency loss takes the transition mode as a constraint condition and retains the spectral information most relevant to the infrared mode from the visible light mode; the calculation process of the semantic consistency loss is as follows:

;

;

Among them, P represents the number of pedestrian categories in the current batch, is the number of visible light/transition images under a single ID, /> It is a fully connected network; /> The hyperparameter weight controls the knowledge propagation rate of strongly and weakly associated sample pairs; is L2 regularization; /> and/> They represent the jth visible light feature and the kth transition feature under the ith identity respectively.

Furthermore, the basic loss consists of identity loss, i.e., cross entropy loss, and triple loss; the calculation process of the visible light modality identity loss is expressed as:

;

Among them, N is the number of visible light images, C is the total number of IDs, The features extracted from the I-th visible light image; is the classifier weight corresponding to the true identity of the sample; the technical process of infrared and transition modes is similar; /> is the corresponding classifier weight;

The triplet loss calculation process is as follows:

;

in, Represents a collection of visible light and transition features; /> With/> is a positive sample pair; With/> is a negative sample pair; /> is the Euclidean distance; /> ; /> is the edge parameter.

Furthermore, in step (5), the person re-identification network is updated by gradient back propagation; the calculation process of the cascade aggregation loss is as follows:

;

in, and/> Respectively represent the visible light and infrared feature centers of the i-th identity; /> and/> Respectively represent the visible light and infrared semantic centers of the ith identity; P is the total number of IDs in the current batch; /> The features extracted from the jth infrared image in the i-th ID in the current batch.

The visible light infrared pedestrian re-identification system based on spectral information filtering of the present invention comprises:

Obtaining preprocessing module: used to obtain raw data, divide it into training set, validation set and test set, and perform preprocessing;

Cross-modal image pair module: used to randomly form cross-modal image pairs from batch training samples processed by the pre-processing module;

Extraction module: used to build a three-branch person re-identification network based on PyTorch and set training parameters. The original training samples and synthesized transition images are used as network input to extract pedestrian representations.

V-T stage module: used to divide the training period into two stages: V-T and V-I. When in the V-T stage, the semantic consistency loss is calculated to update the network weights, and the transition mode is used as a filtering condition to retain the spectral information most relevant to the infrared mode from the visible light mode;

V-I stage module: used to calculate the cascade aggregation loss when in the V-I stage, update the network weights, directly achieve modal alignment between visible light and infrared modalities, and extract modal shared representations; during the training process, the verification set is used to verify the accuracy of the algorithm and save the network weights with the optimal accuracy.

An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded into the processor, implements any one of the visible light infrared pedestrian re-identification methods based on spectral information filtering.

A storage medium described in the present invention stores a computer program, characterized in that when the computer program is executed by a processor, any one of the visible light infrared pedestrian re-identification methods based on spectral information filtering is implemented.

Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it uses deep learning to identify and match pedestrian identities, and has a high recognition rate, which can save a lot of time and labor costs. In addition, the present invention does not add additional model complexity, and only uses global features to achieve good performance, and has low requirements on hardware and computing speed during deployment.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an overall flow chart of the present invention;

FIG2 is a network architecture diagram of a visible light infrared pedestrian re-identification method based on spectral information filtering provided by the present invention;

FIG3 is a schematic diagram of transition mode generation according to the present invention;

FIG4 is a schematic diagram of the two-stage training loss of the present invention;

FIG5 is a flow chart of model training of the present invention.

Detailed ways

The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

As shown in FIGS. 1-5 , an embodiment of the present invention provides a visible light infrared pedestrian re-identification method based on spectral information filtering, comprising the following steps:

(1) Obtain the original data, divide it into training set, validation set and test set and perform preprocessing. The original data set is the public SYSU-MM01 or RegDB. The preprocessing includes: cropping and scaling the training samples to 288x144 pixels, and increasing the diversity of samples by horizontal flipping and random erasing. Finally, the channel mean and standard deviation calculated from the ImageNet dataset are used for standardization.

(2) Randomly group the batch training samples processed in step (1) into cross-modal image pairs; specifically, first randomly select one of the red, green, and blue color channels of the visible light image and expand it into three channels; then divide the processed visible light image and the original infrared image horizontally into multiple parts, and each part retains one of the two modes with the same probability; finally, splice them along the height dimension of the image; each image pair generates a transition image; the formula is as follows:

Suppose we take out the visible light image from the image pair Randomly select one of the three channels of red, green and blue and expand it into three channels, then transform the visible light image and the original infrared image/> The horizontal division is part; then:

;

in, 、/> 、/> Respectively represent the red, green and blue channels in the visible light image, /> and/> For random selection and expansion operations;

;

in, and/> An array of horizontal stripes of the image. Indicates a random selection operation on array elements to obtain a new array; /> It is the stitching operation of the image height dimension.

(3) Based on PyTorch, a three-branch person re-identification network is built and training parameters are set. The original training samples and the synthesized transition images are used as network input to extract pedestrian representations. The details are as follows: Select the ResNet50 model pre-trained on ImageNet as the feature extractor, and then fine-tune the network on the SYSU-MM01 or RegDB dataset; Among them, ResNet50 has a total of five stages. The first stage is replicated three times as the input port of the visible light, transition and infrared modes respectively, and the last four stages are modality shared to form a three-branch pedestrian re-identification network; The visible light and infrared images read in step (1) and the transition images synthesized in step (2) are input into the pedestrian re-identification network to extract pedestrian features.

(4) The training period is divided into two stages, V-T and V-I. When in the V-T stage, the semantic consistency loss is calculated to update the network weights, and the transition mode is used as a filtering condition to retain the spectral information most relevant to the infrared mode from the visible light mode; specifically, the pedestrian features extracted in step (3) and the identity tag information read in step (1) are used to calculate the joint training loss composed of the basic loss and the semantic consistency loss, and the pedestrian re-identification network is updated through gradient back propagation; the semantic consistency loss uses the transition mode as a constraint condition to retain the spectral information most relevant to the infrared mode from the visible light mode; the calculation process of the semantic consistency loss is as follows:

;

;

Among them, P represents the number of pedestrian categories in the current batch, is the number of visible light/transition images under a single ID, /> It is a fully connected network; /> The hyperparameter weight controls the knowledge propagation rate of strongly and weakly associated sample pairs; is L2 regularization; /> and/> They represent the jth visible light feature and the kth transition feature under the ith identity respectively.

The basic loss consists of identity loss, i.e., cross entropy loss, and triple loss; the calculation process of the visible light modality identity loss is expressed as:

;

Among them, N is the number of visible light images, C is the total number of IDs, The features extracted from the I-th visible light image; is the classifier weight corresponding to the true identity of the sample; the technical process of infrared and transition modes is similar; /> is the corresponding classifier weight;

The triplet loss calculation process is as follows:

;

in, Represents a collection of visible light and transition features; /> With/> is a positive sample pair; With/> is a negative sample pair; /> is the Euclidean distance; /> ; /> is the edge parameter.

(5) When in the V-I stage, the cascade aggregation loss is calculated, the network weights are updated, the modal alignment is directly achieved between the visible light and infrared modalities, and the modal shared representation is extracted; during the training process, the accuracy of the algorithm is verified using the validation set, and the network weights with the best accuracy are saved. Among them, the pedestrian re-identification network is updated through gradient back propagation; the calculation process of the cascade aggregation loss is as follows:

;

in, and/> Respectively represent the visible light and infrared feature centers of the i-th identity; /> and/> Respectively represent the visible light and infrared semantic centers of the ith identity; P is the total number of IDs in the current batch; /> The features extracted from the jth infrared image in the i-th ID in the current batch.

The initial learning rate of the present invention is set to 0.01, and then linearly increased to 0.1 after the 10th epoch. After that, the learning rate is reduced by 0.1 times at the 20th and 60th epochs respectively. A batch contains 64 images, of which 4 visible light images and 4 infrared images are randomly selected from 8 identities. The total training duration is 110 epochs, of which the first 100 epochs are used for training in the V-T stage and the last 10 epochs are used for training in the V-I stage. We use SGD as the optimizer, with weight decay set to 0.0005 and momentum set to 0.9.

Excellent performance was achieved on two mainstream visible light infrared pedestrian re-identification datasets, SYSU-MM01 and RegDB. The comparison results with other methods are shown in Table 1:

Table 1: Performance comparison of this method with other visible light infrared pedestrian re-identification methods

The embodiment of the present invention provides a visible light infrared pedestrian re-identification system based on spectral information filtering, comprising:

Obtaining preprocessing module: used to obtain raw data, divide it into training set, validation set and test set, and perform preprocessing;

Cross-modal image pair module: used to randomly form cross-modal image pairs from batch training samples processed by the pre-processing module;

Extraction module: used to build a three-branch person re-identification network based on PyTorch and set training parameters. The original training samples and synthesized transition images are used as network input to extract pedestrian representations.

V-T stage module: used to divide the training period into two stages: V-T and V-I. When in the V-T stage, the semantic consistency loss is calculated to update the network weights, and the transition mode is used as a filtering condition to retain the spectral information most relevant to the infrared mode from the visible light mode;

V-I stage module: used to calculate the cascade aggregation loss when in the V-I stage, update the network weights, directly achieve modal alignment between visible light and infrared modalities, and extract modal shared representations; during the training process, the verification set is used to verify the accuracy of the algorithm and save the network weights with the optimal accuracy.

An embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded into the processor, implements any one of the visible light infrared pedestrian re-identification methods based on spectral information filtering.

An embodiment of the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the visible light infrared pedestrian re-identification methods based on spectral information filtering.

Claims (10)

1. The visible light infrared pedestrian re-identification method based on spectral information filtering is characterized by comprising the following steps of:

(1) Obtaining original data, dividing a training set, a verification set and a test set, and preprocessing;

(2) Randomly forming cross-modal image pairs from the batch training samples processed in the step (1);

(3) Setting up a three-branch pedestrian re-recognition network based on PyTorch, setting training parameters, taking an original training sample and a synthesized transition picture as network input, and extracting pedestrian characterization;

(4) Dividing the training period into two stages of V-T and V-I, calculating the semantic consistency loss updating network weight when the training period is in the V-T stage, taking a transition mode as a filtering condition, and reserving spectrum information most relevant to an infrared mode from a visible light mode;

(5) When the method is in the V-I stage, the cascade aggregation loss is calculated, the network weight is updated, the mode alignment is directly realized between the visible light and the infrared modes, and the mode sharing representation is extracted; in the training process, the accuracy of the algorithm is verified by using a verification set, and the network weight with optimal accuracy is saved.

2. The method for re-identifying visible infrared pedestrians based on spectral information filtering according to claim 1, wherein in the step (1), the original dataset is a published SYSU-MM01 or RegDB; the pretreatment comprises the following steps: training samples were cut and scaled to 288x144 pixels and randomly erased by horizontal flipping to increase sample diversity, and finally normalized using channel mean and standard deviation calculated from the ImageNet dataset.

3. The visible infrared pedestrian re-identification method based on spectral information filtering according to claim 1, wherein the step (2) is specifically as follows: firstly, randomly selecting one of red, green and blue color channels of a visible light image and expanding the red, green and blue color channels into three channels; then horizontally dividing the processed visible light image and the original infrared image into a plurality of parts, wherein each part keeps one of two modes with the same probability; finally, splicing along the height dimension of the image; generating a transition image for each image pair; the formula is as follows:

Setting visible light image in extracted image pair Randomly selecting one of the three channels of red, green and blue, expanding the three channels into three channels, and then transforming the visible light image/>And original infrared image/>Level division into/>A plurality of sections; then:

；

wherein, 、/>、/>Respectively represent red, green and blue channels in the visible light picture,/>AndFor random selection and expansion operations;

；

wherein, And/>Is an array formed by horizontal stripes of the picture,Representing the random selection operation of the array elements to obtain a new array; /(I)And (5) splicing the picture in the height dimension.

4. The visible infrared pedestrian re-identification method based on spectral information filtering according to claim 1, wherein the step (3) is specifically as follows: the ResNet model pre-trained on ImageNet was selected as the feature extractor, and then the network was trimmed on SYSU-MM01 or RegDB dataset; the ResNet stages are five in total, the first stage is duplicated for three times to be used as input ports of visible light, transition and infrared modes respectively, and the last four stages are mode sharing to form a three-branch pedestrian re-identification network; and (3) inputting the visible light and infrared pictures read in the step (1) and the transition pictures synthesized in the step (2) into a pedestrian re-identification network to extract pedestrian characteristics.

5. The visible infrared pedestrian re-identification method based on spectral information filtering of claim 1, wherein the step (4) is specifically as follows: calculating joint training loss consisting of basic loss and semantic consistency loss by utilizing the pedestrian characteristics extracted in the step (3) and the identity tag information read in the step (1), and updating a pedestrian re-recognition network through gradient back propagation; the semantic consistency loss takes a transition mode as a constraint condition, and retains spectrum information most relevant to an infrared mode from a visible light mode; the calculation process of the semantic consistency loss is as follows:

；

；

wherein P represents the number of pedestrian categories in the current batch, For the number of visible/transitional pictures under a single ID,Is a fully connected network; /(I)Controlling knowledge propagation rates of strong correlation and weak correlation sample pairs for the super-parameter weights; /(I)Regularization for L2; /(I)And/>Respectively representing the jth visible light characteristic and the kth transition characteristic under the ith identity.

6. The method for re-identifying visible infrared pedestrians based on spectral information filtering according to claim 5, wherein the basic loss consists of identity loss, i.e. cross entropy loss and triplet loss; the calculation process of the visible light modal identity loss is expressed as follows:

；

wherein N is the number of visible light pictures, C is the total number of IDs, Extracting the obtained characteristics of the I-th visible light picture; /(I)Classifier weights corresponding to the true identities of the samples; /(I)The classifier weights are corresponding;

The triplet loss calculation process is as follows:

；

wherein, Representing a set of visible light and transition features; /(I)And/>Is a positive sample pair; /(I)And (3) withIs a negative sample pair; /(I)Is the Euclidean distance; /(I)；/>Is an edge parameter.

7. The method for re-identifying visible infrared pedestrians based on spectral information filtering according to claim 1, wherein in the step (5), the pedestrian re-identification network is updated by gradient back propagation; the calculation process of the cascade aggregation loss is as follows:

；

wherein, And/>Visible and infrared feature centers respectively representing the ith identity; /(I)And/>Visible and infrared semantic centers respectively representing the ith identity; p is the total number of IDs in the current batch; /(I)And extracting the obtained characteristics for the j-th infrared picture in the i-th ID in the current batch.

8. A visible infrared pedestrian re-identification system based on spectral information filtering, comprising:

and (3) an acquisition pretreatment module: the method comprises the steps of obtaining original data, dividing a training set, a verification set and a test set, and preprocessing;

cross-modality image pair module: the system comprises a preprocessing module, a cross-modal image acquisition module and a cross-modal image acquisition module, wherein the preprocessing module is used for preprocessing the acquired training samples;

And an extraction module: the method is used for building a three-branch pedestrian re-recognition network based on PyTorch, setting training parameters, taking an original training sample and a synthesized transition picture as network input, and extracting pedestrian characterization;

V-T stage module: the training period is divided into two stages, namely a V-T stage and a V-I stage, when the training period is in the V-T stage, the semantic consistency loss is calculated to update the network weight, the transition mode is used as a filtering condition, and the spectrum information most relevant to the infrared mode is reserved from the visible light mode;

V-I stage module: the method is used for calculating cascading aggregation loss when the system is in a V-I stage, updating network weight, directly realizing modal alignment between visible light and infrared modes, and extracting modal sharing representation; in the training process, the accuracy of the algorithm is verified by using a verification set, and the network weight with optimal accuracy is saved.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the computer program when loaded into the processor implements a method for visible infrared pedestrian re-identification based on spectral information filtering according to any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the visible infrared pedestrian re-recognition method based on spectral information filtering according to any one of claims 1-7.