CN116778530A - Cross-appearance pedestrian re-identification detection method based on generation model - Google Patents
Cross-appearance pedestrian re-identification detection method based on generation model Download PDFInfo
- Publication number
- CN116778530A CN116778530A CN202310898793.2A CN202310898793A CN116778530A CN 116778530 A CN116778530 A CN 116778530A CN 202310898793 A CN202310898793 A CN 202310898793A CN 116778530 A CN116778530 A CN 116778530A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- appearance
- image
- model
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 13
- 238000012549 training Methods 0.000 claims abstract description 28
- 238000000034 method Methods 0.000 claims abstract description 18
- 238000003708 edge detection Methods 0.000 claims abstract description 9
- 230000011218 segmentation Effects 0.000 claims abstract description 9
- 230000004927 fusion Effects 0.000 claims abstract description 7
- 238000004458 analytical method Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 description 12
- 241000282414 Homo sapiens Species 0.000 description 11
- 230000000694 effects Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 230000006872 improvement Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 101000611614 Homo sapiens Proline-rich protein PRCC Proteins 0.000 description 1
- 102100040829 Proline-rich protein PRCC Human genes 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003042 antagnostic effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000037237 body shape Effects 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
- G06V40/25—Recognition of walking or running movements, e.g. gait recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Biomedical Technology (AREA)
- Psychiatry (AREA)
- Life Sciences & Earth Sciences (AREA)
- Social Psychology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a cross-appearance pedestrian re-recognition detection method based on a generation model, which belongs to the technical field of pedestrian re-recognition and comprises the following steps of: in the generation model, body type characteristics and appearance characteristics are exchanged among different pedestrian images, and a new pedestrian image is generated; the RGB image of the pedestrian image is subjected to a pre-trained edge detection network and a pedestrian semantic segmentation network to obtain a pedestrian profile sketch and a pedestrian analytic graph; the pedestrian contour sketch, the RGB image and the pedestrian analytic graph of the pedestrian image of the pedestrian data set are input into the same backbone network to extract the characteristics, and the characteristics are fused and then the reasoning training is carried out. The method comprises the steps of carrying out appearance dimension enhancement on a pedestrian image by using a generated model, and introducing the generated pedestrian image into a training stage of the model; the three modes use the same backbone network to extract the characteristics, the key characteristics of different appearances of the same pedestrian are guided to be learned by the model through fusion and reasoning, and the model has more robust performance in a cross-appearance scene.
Description
Technical Field
The invention belongs to the technical field of pedestrian re-recognition, and particularly relates to a cross-appearance pedestrian re-recognition detection method based on a generation model.
Background
At present, the application scene of the computer vision technology is very wide. The face recognition technology is superior to human beings, is widely applied to industries such as industry, medical treatment, education and the like, and the eyes in academic circles and industry gradually turn to a subject with scientific research significance and application value, namely pedestrian re-recognition. In practical scenes such as transportation, industrial manufacturing and the like, most cases are the case of blurring a human face and even not containing the human face, so that the human face recognition effect is very limited.
The Re-identification (Re-ID) aims at giving a monitoring pedestrian image to search other images of the pedestrian under cross equipment, and the pedestrian Re-identification technology can make up for the failure of face recognition and the visual limitation of a fixed camera and is applied to the fields of video monitoring, intelligent security, intelligent life and the like. Because Re-ID needs to find objects in images and videos across devices, the resolution and the positions of the devices are different, the coverage areas of the devices are not overlapped, so that the lack of coherent information is caused, the illumination, the background and the shielding of scenes are different, the posture and the appearance of the objects are changed, and the Re-recognition technology of pedestrians is greatly challenged.
A pedestrian re-recognition method based on deep learning feature learning. Global features: the feature learning is mainly carried out by using global images of the whole body, and common improvement ideas include an Attention mechanism, multi-scale fusion and the like; local features: characteristic learning is carried out by utilizing a local image area, such as a partial structure of a pedestrian or simple vertical area division, and finally, the local characteristics are aggregated into final pedestrian characteristics for recognition; auxiliary characteristics: enhancing the effect of feature learning by using some auxiliary information, such as semantic information, visual angle information, domain information, GAN generated information, data enhancement and the like; specific network design: by utilizing the characteristics of Re-ID tasks, a plurality of network structures with fine granularity, multiscale and the like are designed, so that the Re-ID task is more suitable for the Re-ID scene.
The pedestrian re-recognition method based on the deep learning measurement learning comprises the following steps: mainly comprising the design of different types of loss functions and the improvement of sampling strategies. Identity loss, namely taking a training process of Re-ID as an image classification problem, taking different pictures of the same pedestrian as a category, and commonly having a Softmax cross entropy loss function; verification loss: training Re-ID is taken as an image matching problem, whether the training belongs to the same pedestrian or not is subjected to classification learning, and a common contrast loss function and a classification loss function are adopted; triplet loss: training Re-ID is taken as an image retrieval problem, and the characteristic distance of the same pedestrian picture is smaller than that of different pedestrians, and various improvements thereof are carried out; improvement of training strategy: adaptive sampling and different weight allocation strategies.
The pedestrian re-identification method across the appearance comprises the following steps: along with the release of cross-appearance pedestrian re-identification data sets, researchers have proposed cross-appearance pedestrian re-identification methods applied to these data sets, which mostly forgo clothing-related appearance features. The RF-Reid is an end-to-end model for pedestrian re-identification, and takes a radio frequency track as input, and features are extracted from the track, so that the model can obtain accurate human body contours for identification. In PRCC, researchers use Angle Specific Extractors (ASE) to extract fine-grained angle specific discrimination features by varying the sampling range of the SPT and to contribute a multi-stream network to aggregate multi-granularity features. The BC-Net uses the clothing template to search the candidate pedestrian image by using the double-branch network, so that the biological characteristics and clothing characteristics are effectively fused. CASE-Net utilizes a cross-appearance, antagonistic learning strategy to extract body type features and structure body type through image generation of posture changes. FSAM proposes a dual stream framework that learns fine-grained body-type related information from a body-type stream and transfers it into an appearance stream to supplement garment-independent feature information in appearance features.
Early research of the pedestrian re-recognition model mainly focuses on manually designing features and learning better similarity measurement, and with the development of deep learning, better features and learning better similarity habit measurement are automatically extracted through a neural network, so that the pedestrian re-recognition performance is greatly improved. However, simplification of the application scene in these studies is performed, and the pedestrian's clothing is not changed, assuming that the search is performed only in a short time, so that the pedestrian re-recognition model focuses mostly on the color and texture features of the pedestrian's clothing, which limits the application scene of the model. Due to human and time constraints, existing cross-appearance pedestrian re-identification datasets suffer from the following drawbacks: 1. the presently existing cross-appearance datasets are small in size and insufficient to support training of deep learning models. 2. The bias of the background of the data set and the like is greatly different from the real world, which can cause domain adaptation problems of the model. 3. The time interval of source video acquisition is short, no obvious appearance change exists, and appearance difference is difficult to observe. In a real scene, people can change clothes according to factors such as time weather, and the assumption that appearance of the pedestrian is consistent in a short period is not applicable. The new scene also provides new challenges for the pedestrian re-recognition model, and the model is required to be independent of the appearance characteristics related to clothes, and focuses on robust characteristic information such as hairstyles, body types and the like of people.
Disclosure of Invention
Aiming at the technical problems of poor long-time cross-appearance pedestrian re-recognition performance, large clothing change difference and lack of related data sets in the prior art, the invention provides a cross-appearance pedestrian re-recognition detection method based on a generation model, which uses the generation model to strengthen the appearance dimension of a pedestrian image and introduces the generated pedestrian image into a training stage of the model; meanwhile, a multi-mode pedestrian image is introduced, besides the traditional RGB image, two images with smaller appearance correlation, namely a pedestrian contour sketch and a pedestrian analytic image, are additionally added, and the three modes use the same backbone network to extract features, and are fused for reasoning.
The technical scheme adopted by the invention is as follows: a cross-appearance pedestrian re-identification detection method based on a generated model comprises the following steps:
step 1: inputting pedestrian images of a pedestrian data set into a generation model, respectively obtaining body type characteristics and appearance characteristics of the pedestrian images by different encoders of the generation model, exchanging the body type characteristics and the appearance characteristics among different pedestrian images, generating new pedestrian images through a generator, identifying tag information of the new pedestrian images by matching with a discriminator, and storing the new pedestrian images and the tag information thereof into the pedestrian data set;
step 2: taking RGB images of all pedestrian images of the pedestrian data set as input, respectively extracting contour information and semantic segmentation information of pedestrians through a pre-trained edge detection network and a pre-trained pedestrian semantic segmentation network to obtain a pedestrian contour sketch and a pedestrian analysis drawing, and storing the pedestrian contour sketch and the pedestrian analysis drawing in the pedestrian data set;
step 3: inputting the pedestrian profile sketch, the RGB image and the pedestrian analysis drawing of the pedestrian image of the pedestrian data set into the same backbone network to extract the characteristics to obtain f Contour 、f RGB And f Parsing Feature fusion to obtain f Conb And then carrying out reasoning training, and carrying out cross-appearance pedestrian re-identification detection on the trained backbone network.
Further, in the step 1, after the pedestrian image is input into the generation model, appearance characteristics and body type characteristics are respectively obtained through an appearance encoder and a body type encoder, the appearance characteristics and the body type characteristics of the same pedestrian image are input into a generator to obtain a self-reconstruction pedestrian image, and a discriminator discriminates the tag information of the self-reconstruction pedestrian image; and the self-reconstruction pedestrian image acquires appearance characteristics and body type characteristics through the appearance encoder and the body type encoder respectively, the appearance characteristics of the self-reconstruction pedestrian image and the body type characteristics of another self-reconstruction pedestrian image are input into the generator to obtain a new pedestrian image, the identifier identifies the label information of the new pedestrian image, and the new pedestrian image and the label information are stored in the pedestrian data set.
Further, in step 1, the appearance encoder is a feature extraction network of the pedestrian re-recognition network; the body type encoder is a network that introduces a pyramid structure of interest to different scales.
Further, when the generated model is trained, the appearance encoder uses an identity loss function to conduct constraint, and the body type encoder uses an identity tag loss function to conduct training of the model.
Further, in step 2, the edge detection network uses an edge detection data set to complete pre-training; the pedestrian semantic segmentation network utilizes the LIP data set to complete pre-training.
Further, in step 3, the f Contour 、f RGB And f Parsing The fusion mode of (a) is weighted splicing.
Further, in step 3, the backbone network adopts a Densenet121.
Further, in step 3, the loss of the backbone network includes the f Contour 、f RGB And f Parsing TriHard loss of (2)F is as follows Conb Identity loss of->Appearance loss->The formula is as follows:
in the above formula (1), P and K respectively represent the number of pedestrians and the number of corresponding images in one batch, and an alpha interval parameter alpha and a distance measure D are given for the kth image of the kth person in the batch Obtained byK' th image for p-th person in lot +.>Obtained->K 'th image for p' th person in batch +.>Obtained->Restricting its characteristics to the most difficult positive samples +.>And the characteristics of the most difficult negative samples->Is a distance of (2); in the above formulas (2) and (3), the image x is given i Identity tag y i And appearance label z i ，p(y i |x i ) And p (z) i |x i ) Respectively representing the model to recognize the image as identity tag y i And appearance label z i Is a probability distribution of (c).
Compared with the prior art, the invention has the following beneficial effects:
1. according to the invention, the generation model is introduced, on one hand, data enhancement can be performed through the generation model, and on the other hand, the quality of the pedestrian data set is improved by increasing the diversity of pedestrians in the appearance dimension under the condition of a limited pedestrian data set; on the other hand, the appearance encoder and the body type encoder of the generated model are used for realizing separation of two types of features related to appearance and robust to cross-appearance, the characteristics extracted by the body type encoder and the other characteristics are fused, and the adaptability of the model to appearance change is improved, so that the pedestrian re-recognition model with better performance is obtained. The image generated by the generated model can fill the gap of the original data set on the distribution curve, the illumination or contrast of the generated image and the original data set are more similar, the generated image is close to the actual test set, and compared with the combination of multiple data sets, the generated image is more close to the test set, so that the result of the model is more likely to be improved.
2. Because the traditional pedestrian re-recognition model is excessively dependent on appearance characteristics, the invention adopts three pedestrian images with different modes, not only maintains the adaptability of the model to the traditional pedestrian re-recognition task, but also reduces the dependence of the model recognition on the appearance, and the characteristics for reasoning are more suitable for scenes crossing the appearance. According to the invention, the pedestrian re-recognition model is fused with multi-mode information, and the target retrieval across the appearance is improved by considering two appearance independent clues, namely the contour feature and the human body analysis feature, so that the model is guided to learn key features of the same pedestrian with different appearances, and has more robust performance under a scene across the appearances.
Drawings
FIG. 1 is a flow chart of an embodiment of the present invention;
FIG. 2 is a schematic diagram of a generative model according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an image extraction and pedestrian re-recognition model according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail below with reference to the drawings and the specific embodiments, so that those skilled in the art can better understand the technical solutions of the present invention.
The embodiment of the invention provides a cross-appearance pedestrian re-identification detection method based on a generation model, which is shown in fig. 1 and comprises the following steps:
step 1: the number of images of the pedestrian data set has a great influence on the performance of the deep learning model, and the number of appearances of pedestrians is compared with the posture or the angle according to the arrangement and the related scale of the common pedestrian data set, so that the two appearances at present cannot support the model to be optimized. Other clothing images of the pedestrian are generated by combining the pedestrian with various appearances and the like by using the generation model, so that a large number of appearances of a single pedestrian are formed, and the generation model is guided to learn robust features of the pedestrian.
The characteristics of the cross-appearance pedestrian image are generally divided into two types, one type is the characteristics of the pedestrian, such as height, body shape and the like, the characteristics are irrelevant to the appearance of the pedestrian, such as clothing and the like, and the characteristics are expected characteristic information of the model; the other is the appearance information of the color, texture and the like of the clothing, and the characteristics of the other type occupy a larger proportion in the conventional pedestrian re-recognition. In this embodiment, the pedestrian image of the pedestrian data set is input into the generation model, different encoders of the generation model are used to obtain the body type feature and the appearance feature of the pedestrian image respectively, the body type feature and the appearance feature are exchanged between different pedestrian images, the images of different pedestrians and different appearances of the same pedestrian are exchanged, then a new pedestrian image is generated through the generator, the label information of the new pedestrian image is identified by matching with the identifier, the process achieves the purpose of separating the two features, and finally the new pedestrian image and the label information thereof are stored in the pedestrian data set. The new pedestrian image is generated to enrich the diversity of the pedestrian image, and the quality of the pedestrian data set can be improved by adding the generated pedestrian image into the training stage of the model, so that the capability of the model for processing the cross-appearance problem is improved. As shown in fig. 2, the specific procedure of data enhancement of the pedestrian data set by using the generative model is as follows: after the pedestrian data set is input into the generation model, the appearance characteristic and the body type characteristic are respectively obtained through the appearance encoder and the body type encoder, the appearance characteristic and the body type characteristic of the same pedestrian image are input into the generator to obtain a self-reconstruction pedestrian image, and the identifier identifies the label information of the self-reconstruction pedestrian image; the appearance characteristics and the body type characteristics of the self-reconstruction pedestrian image are acquired through the appearance encoder and the body type encoder respectively, the appearance characteristics of the self-reconstruction pedestrian image and the body type characteristics of another self-reconstruction pedestrian image are input into the generator to obtain a new pedestrian image, the identifier identifies the label information of the new pedestrian image, and the new pedestrian image and the label information thereof are stored in the pedestrian data set.
The body type encoder and the appearance encoder of the generative model are constrained using different loss functions, so that the two encoders pay attention to the characteristics of different parts of the pedestrian image. Based on the related knowledge of the appearance information such as pedestrian clothing and the like, which is focused by the traditional pedestrian re-recognition algorithm, the appearance encoder uses the characteristic extraction network of the common pedestrian re-recognition network, so that the common identity loss function is used for constraint. The body type encoder uses a network which is focused on pyramid structures with different scales, and the traditional pedestrian data set does not contain appearance changes in the pedestrian image, so that the identity tag is consistent with the appearance tag; in the embodiment, the identity tag of the pedestrian is introduced into the training of the generation model, and the training of the model is guided by using the identity tag loss function. The discriminator of the generated model continuously improves the quality of the generated image by judging the true or false of the generated image against the generator.
The quality of the training data is not dependent on the quantity, and the representativeness of the data should be more emphasized so that the model can learn more robust features. The pedestrian images generated by the generated model are derived from the training set, and are different from the training set, so that the pedestrian images can fill in the gaps on the data distribution curve and enrich the appearance dimension of the pedestrian images. In the training images of the selected generation model, the long-time appearance-crossing characteristic is combined, the input of the generation model is adjusted according to the time span, and the pedestrian images with large appearance differences are selected for training and generation, so that obvious appearance differences are realized.
Step 2: the outline of the pedestrian is mainly the edge information of the pedestrian, and the important information of the body type of the pedestrian is included; the human body analysis chart is mainly used for obtaining the information of the local area of the pedestrian and is often used as auxiliary information for pedestrian alignment. The outline information and the human body analytic graph of pedestrians are robust to the appearance change to a certain extent, and the pedestrian information robust to the appearance change can be provided for the network.
The process of extracting the contour of the pedestrian and the human body analytic graph is a priori knowledge extraction stage of the model. Before training of the model, RGB images of all pedestrian images of the pedestrian data set are used as input, the contour information and the semantic segmentation information of the pedestrians are respectively extracted through an edge detection network pre-trained on the edge detection data set and a pedestrian semantic segmentation network pre-trained on the LIP data set to obtain a pedestrian contour sketch and a pedestrian analysis drawing, and all images are stored in the pedestrian data set according to an original path, as shown in fig. 3.
Step 3: as shown in fig. 3, the pedestrian profile sketch, RGB image and pedestrian resolution of the pedestrian image of the pedestrian dataset are input with the same backbone extraction features to obtain f Contour 、f RGB And f Parsing Feature fusion to obtain f Conb And then carrying out reasoning training, and carrying out cross-appearance pedestrian re-identification detection on the trained backbone network.
The contour information and the human body analysis chart of the pedestrians cannot be completely used as the features of the re-recognition of the pedestrians, so that besides introducing two modes into a re-recognition network of the pedestrians, the RGB images of the pedestrians are still kept as input, global features of the RGB images and features of other two modes are obtained through a backbone network, the features and the features of the RGB images are mutually supervised and mutually supplemented with more details in the RGB images, and finally the three features are subjected to weighted splicing, so that the dependence on the appearance-related features of the pedestrians in the recognition of the pedestrians is effectively reduced, and the precision of a re-recognition model of the pedestrians is improved.
The pedestrian re-recognition process is generally characterized by two stages of feature extraction and distance measurement, and the quality of a feature extraction network directly influences the quality of a pedestrian re-recognition model. The existing work commonly uses a deep network as a backbone network, and the effect of feature extraction has larger difference due to the arrangement of a network structure and the difference of network layers. In this embodiment, the Densenet121 is selected as a backbone network of the network, and experiments are performed according to different depth networks respectively, so that the scale of the network is gradually increased on the basis of guaranteeing the pedestrian recognition speed, and better pedestrian re-recognition performance is obtained. According to experience, there is generally a phenomenon that in the structure of a deep neural network, as the number of layers of the network is deepened, a better result is generally obtained in the model, but a phenomenon that gradient vanishes gradually occurs to significantly reduce the training effect of the model, while the Densenet121 connects the input of each layer with the output of all the previous layers, so as to alleviate the gradient vanishing problem, connect the shallow layer characteristics and the deep layer characteristics of the pedestrian, and achieve a mode of combining the global characteristics and the fine-grain local characteristics, so that the problem that the multi-scale characteristics are needed for the re-identification of the pedestrian is better adaptive to the problem, as shown in table 1.
Table 1 network structure and output size table of Densenet121
The existing pedestrian re-identification model often adopts a TriHard loss function, and a positive sample with the lowest similarity to the anchor point image and a negative sample with the highest similarity are selected from one batch to calculate a ternary loss function. In order to guide the Densenet121 network to learn the data and the characteristics thereof in three different modes together, the embodiment introduces identity loss, appearance loss and TriHard loss, namely f Contour 、f RGB And f Parsing TriHard loss of f) Conb Identity loss and appearance loss of the product. The 5 losses are adopted to jointly carry out backbone network parameter adjustment, and when parameter adjustment is carried out, the appearance loss needs to be multiplied by a coefficient lambda, 0<λ<1. Identity loss can guide the model to learn the characteristics related to appearance, and identity loss and appearance loss can guide the model to learn the characteristics related to the identity of pedestrians, and TriHard loss enables the model to obtain a more robust effect in classification of difficult samples. Wherein TriHrad loss calculates the similarity of feature vectors in European space, and identity loss and appearance loss calculate vector similarity in cosine space. The formulas for the three loss functions are as follows:
in the above formula (1), P and K respectively represent the number of pedestrians and the number of corresponding images in one batch, and an alpha interval parameter alpha and a distance measure D are given for the kth image of the kth person in the batch Obtained byK' th image for p-th person in lot +.>Obtained->K 'th image for p' th person in batch +.>Obtained->Restricting its characteristics to the most difficult positive samples +.>And the characteristics of the most difficult negative samples->Is a distance of (2); in the above formulas (2) and (3), the image x is given i Identity tag y i And appearance label z i ，p(y i |x i ) And p (z) i |x i ) Respectively representing the model to recognize the image as identity tag y i And appearance label z i Is a probability distribution of (c).
In order to measure the performance of the model for identifying the pedestrian re-recognition based on the cross-appearance of the generated model, in this embodiment, a comparison experiment is performed on the NKUP data set by using the (our) and other SOTA pedestrian re-recognition methods, and the experimental results are shown in table 2. In this embodiment, rank 1 and mAP are used to evaluate the performance of the model, where the former represents the precision of ranking the first pedestrian image in the query list, and the latter is the average precision of all the samples to be queried. The reference model in table 2 is a network model enhanced based on generated model data, and from experimental results, it can be seen that the method provided by the embodiment has significant advantages, especially, the best result is obtained in the scene of recognition of pedestrians across appearances, and the lifting effect of the model in recognition of pedestrians across appearances is verified.
Table 2 comparison of experimental results
The present invention has been described in detail by way of examples, but the description is merely exemplary of the invention and should not be construed as limiting the scope of the invention. The scope of the invention is defined by the claims. In the technical scheme of the invention, or under the inspired by the technical scheme of the invention, similar technical schemes are designed to achieve the technical effects, or equivalent changes and improvements to the application scope are still included in the protection scope of the patent coverage of the invention.
Claims (8)
1. The cross-appearance pedestrian re-identification detection method based on the generated model is characterized by comprising the following steps of:
step 1: inputting pedestrian images of a pedestrian data set into a generation model, respectively obtaining body type characteristics and appearance characteristics of the pedestrian images by different encoders of the generation model, exchanging the body type characteristics and the appearance characteristics among different pedestrian images, generating new pedestrian images through a generator, identifying tag information of the new pedestrian images by matching with a discriminator, and storing the new pedestrian images and the tag information thereof into the pedestrian data set;
step 2: taking RGB images of all pedestrian images of the pedestrian data set as input, respectively extracting contour information and semantic segmentation information of pedestrians through a pre-trained edge detection network and a pre-trained pedestrian semantic segmentation network to obtain a pedestrian contour sketch and a pedestrian analysis drawing, and storing the pedestrian contour sketch and the pedestrian analysis drawing in the pedestrian data set;
step 3: inputting the pedestrian profile sketch, the RGB image and the pedestrian analysis drawing of the pedestrian image of the pedestrian data set into the same backbone network to extract the characteristics to obtain f Contour 、f RGB And f Parsing Feature fusion to obtain f Conb And then carrying out reasoning training, and carrying out cross-appearance pedestrian re-identification detection on the trained backbone network.
2. The method for detecting the cross-appearance pedestrian re-recognition based on the generated model as claimed in claim 1, wherein in the step 1, after the pedestrian image is input into the generated model, appearance characteristics and body type characteristics are respectively obtained through an appearance encoder and a body type encoder, the appearance characteristics and the body type characteristics of the same pedestrian image are input into a generator to obtain a re-constructed pedestrian image, and a discriminator discriminates label information of the re-constructed pedestrian image; and the self-reconstruction pedestrian image acquires appearance characteristics and body type characteristics through the appearance encoder and the body type encoder respectively, the appearance characteristics of the self-reconstruction pedestrian image and the body type characteristics of another self-reconstruction pedestrian image are input into the generator to obtain a new pedestrian image, the identifier identifies the label information of the new pedestrian image, and the new pedestrian image and the label information are stored in the pedestrian data set.
3. The method for detecting the cross-appearance pedestrian re-recognition based on the generated model as claimed in claim 2, wherein in the step 1, the appearance encoder is a feature extraction network of a pedestrian re-recognition network; the body type encoder is a network that introduces a pyramid structure of interest to different scales.
4. A cross-appearance pedestrian re-recognition detection method based on a generative model as in claim 3, wherein the generative model is trained by the appearance encoder constrained using an identity loss function, and the body type encoder directs training of the model using an identity tag loss function.
5. The method for detecting the re-recognition of the pedestrian across the appearance based on the generated model as claimed in claim 1, wherein in the step 2, the edge detection network uses an edge detection data set to complete the pre-training; the pedestrian semantic segmentation network utilizes the LIP data set to complete pre-training.
6. The method for detecting cross-appearance pedestrian re-recognition based on a generated model as claimed in claim 1, wherein in step 3, the f Contour 、f RGB And f Parsing The fusion mode of (a) is weighted splicing.
7. The method for detecting the re-recognition of the pedestrian across the appearance based on the generated model as claimed in claim 1, wherein in the step 3, the backbone network adopts a Densenet121.
8. The method for cross-appearance pedestrian re-recognition detection based on a generative model as recited in claim 1 or 7, wherein in step 3, the loss of the backbone network comprises the f Contour 、f RGB And f Parsing TriHard loss of (2)F is as follows Conb Identity loss of->Appearance loss->The formula is as follows:
in the above formula (1), P and K respectively represent the number of pedestrians and the number of corresponding images in one batch, and an alpha interval parameter alpha and a distance measure D are given for the kth image of the kth person in the batch1.ltoreq.k.ltoreq.K)>K' th image for p-th person in lot +.>1.ltoreq.k'.ltoreq.K)>K 'th image for p' th person in batch +.>1.ltoreq.k'.ltoreq.K)>Constraining its characteristics to the most difficult positive samplesAnd the characteristics of the most difficult negative samples->Is a distance of (2); in the above formulas (2) and (3), the image x is given i Identity tag y i And appearance label z i ，p(y i |x i ) And p (z) i |x i ) Separate tableThe model recognizes the image as an identity tag y i And appearance label z i Is a probability distribution of (c).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310898793.2A CN116778530A (en) | 2023-07-21 | 2023-07-21 | Cross-appearance pedestrian re-identification detection method based on generation model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310898793.2A CN116778530A (en) | 2023-07-21 | 2023-07-21 | Cross-appearance pedestrian re-identification detection method based on generation model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116778530A true CN116778530A (en) | 2023-09-19 |
Family
ID=88013403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310898793.2A Pending CN116778530A (en) | 2023-07-21 | 2023-07-21 | Cross-appearance pedestrian re-identification detection method based on generation model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116778530A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117831081A (en) * | 2024-03-06 | 2024-04-05 | 齐鲁工业大学(山东省科学院) | Method and system for re-identifying clothing changing pedestrians based on clothing changing data and residual error network |
| CN117953539A (en) * | 2024-01-24 | 2024-04-30 | 南京信息工程大学 | Sketch pedestrian re-recognition method adjusted by cross-mode prompt |
-
2023
- 2023-07-21 CN CN202310898793.2A patent/CN116778530A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117953539A (en) * | 2024-01-24 | 2024-04-30 | 南京信息工程大学 | Sketch pedestrian re-recognition method adjusted by cross-mode prompt |
| CN117831081A (en) * | 2024-03-06 | 2024-04-05 | 齐鲁工业大学(山东省科学院) | Method and system for re-identifying clothing changing pedestrians based on clothing changing data and residual error network |
| CN117831081B (en) * | 2024-03-06 | 2024-05-24 | 齐鲁工业大学(山东省科学院) | Method and system for re-identifying clothing changing pedestrians based on clothing changing data and residual error network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tao et al. | Person re-identification by regularized smoothing kiss metric learning | |
| CN111709311B (en) | Pedestrian re-identification method based on multi-scale convolution feature fusion | |
| CN108520226B (en) | Pedestrian re-identification method based on body decomposition and significance detection | |
| CN103632132B (en) | Face detection and recognition method based on skin color segmentation and template matching | |
| CN111666843A (en) | Pedestrian re-identification method based on global feature and local feature splicing | |
| Shi et al. | Real-time traffic light detection with adaptive background suppression filter | |
| CN109359541A (en) | A kind of sketch face identification method based on depth migration study | |
| Rao et al. | Multi-pose facial expression recognition based on SURF boosting | |
| CN110717411A (en) | Pedestrian re-identification method based on deep layer feature fusion | |
| CN116778530A (en) | Cross-appearance pedestrian re-identification detection method based on generation model | |
| CN111178208A (en) | Pedestrian detection method, device and medium based on deep learning | |
| Guo et al. | Improved hand tracking system | |
| CN109447082B (en) | Scene moving object segmentation method, system, storage medium and equipment | |
| Shah et al. | A novel biomechanics-based approach for person re-identification by generating dense color sift salience features | |
| CN112149538A (en) | Pedestrian re-identification method based on multi-task learning | |
| CN113963032A (en) | Twin network structure target tracking method fusing target re-identification | |
| Zhang et al. | A survey on face anti-spoofing algorithms | |
| CN109271932A (en) | Pedestrian based on color-match recognition methods again | |
| CN114782977B (en) | Pedestrian re-recognition guiding method based on topology information and affinity information | |
| CN112990120B (en) | Cross-domain pedestrian re-identification method using camera style separation domain information | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| CN112541421A (en) | Pedestrian reloading identification method in open space | |
| Peng et al. | Presentation attack detection based on two-stream vision transformers with self-attention fusion | |
| CN105893967B (en) | Human behavior classification detection method and system based on time sequence retention space-time characteristics | |
| CN117095471A (en) | Face counterfeiting tracing method based on multi-scale characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |