CN112036250B - Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention - Google Patents
Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention Download PDFInfo
- Publication number
- CN112036250B CN112036250B CN202010772245.1A CN202010772245A CN112036250B CN 112036250 B CN112036250 B CN 112036250B CN 202010772245 A CN202010772245 A CN 202010772245A CN 112036250 B CN112036250 B CN 112036250B
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- image
- feature
- neighborhood
- identified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention provides a pedestrian re-identification method, a system, a medium and a terminal based on neighborhood cooperative attention; the method comprises the following steps: performing primary processing on the obtained pedestrian image to generate a pedestrian image to be identified after the primary processing; performing feature inference on the image of the pedestrian to be recognized to acquire a first vector feature of the pedestrian to be recognized on the image of the pedestrian to be recognized; performing feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image; calculating the similarity between the first vector feature and the second vector feature so as to realize pedestrian re-identification of the pedestrian to be identified according to the similarity; the invention integrates the overall characteristics of the image and the characteristics of the local area, combines the neighborhood local characteristics of different scales through the context neighborhood connection of the local characteristics, better notices the local detail characteristics of different scales of the pedestrian, realizes the accurate matching of the images of the same pedestrian and improves the accuracy of pedestrian re-identification.
Description
Technical Field
The invention belongs to the technical field of pedestrian re-identification, and particularly relates to a pedestrian re-identification method, a system, a medium and a terminal based on neighborhood cooperative attention.
Background
Pedestrian Re-Identification (Re-Identification) is one of very challenging tasks in the field of computer vision, namely, the pedestrian Re-Identification is to search a given pedestrian from a large number of pedestrian images acquired by different cameras, and has important value and significance in the fields of commercial passenger flow and security protection at present; in the field of commercial security, a high-quality face picture cannot be obtained in a monitoring video due to the camera resolution and the shooting angle, when face recognition fails, Re-ID becomes a very important retrieval technology, and can quickly scout suspicious personnel, accurately position and identify the suspicious personnel and other monitored objects, quickly position and find lost or abducted children in public areas with dense people flows, and has great significance for maintaining the public safety of markets; in the commercial customer flow field, through the Re-ID technology, a lot of unstructured information of customers and stores can be converted into structured information to be presented, behavior tracks, store-arriving times, residence time and the like of the customers of the stores are obtained, and merchants are helped to mine more customer information and achieve more commercial values.
At the present stage, due to the scene complexity of a market monitoring video, the main challenges of the pedestrian Re-ID come from the great changes of the pedestrian such as posture, occlusion, clothes, background confusion, detection failure and the like, and the vigorous development of a deep convolutional network improves the performance of the Re-ID to a new level; in recent years, the Re-ID method based on attention mechanism is proposed to make the technology have breakthrough progress in the field of commercial passenger flow, at present, the Re-ID technology based on attention mechanism divides the global feature of the pedestrian into local features with fixed spatial position and different levels in a multi-granularity way, and the two are combined to improve the response to the local information of the pedestrian during identification, such as the classic structures of MGN, PCB and the like, but in the practical market passenger flow application, due to the difference of camera equipment, the shooting scene has randomness, the appearance of the pedestrian is easily affected by the external factors such as wearing, size and posture, viewing angle and the like, especially the problem of blocking, a pedestrian can be partially blocked by other people or objects, a whole body image can not be generated frequently, and when some local information is blocked, a good context local response can not be achieved, but the effect of pedestrian Re-identification can be affected, moreover, there are differences of different scales between different pedestrians, and if the segmentation is performed only according to a fixed scale, the identification of local features of different scales cannot be satisfied, as shown in fig. 10.
As can be seen from fig. 10, two groups of pedestrians have differences in local features of different sizes, for example, the first group of data has differences in small-sized shoulder features and large-sized trousers features, and if only the segmented local features are used for learning, local feature difference information of neighborhood context cannot be obtained, so that the identity of the pedestrian cannot be well judged, and the identification accuracy is affected.
Disclosure of Invention
In view of the above drawbacks of the prior art, an object of the present invention is to provide a pedestrian re-identification method, system, medium and terminal based on neighborhood synergistic attention, for solving the problem of the prior art that the accuracy of pedestrian re-identification is reduced due to the influence of external factors.
In order to achieve the above objects and other related objects, the present invention provides a pedestrian re-identification method based on neighborhood cooperative attention, including the steps of: performing primary processing on the obtained pedestrian image to generate a pedestrian image to be identified after the primary processing; performing feature inference on the pedestrian image to be identified to acquire a first vector feature of the pedestrian to be identified on the pedestrian image to be identified; performing feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image; and calculating the similarity between the first vector feature and the second vector feature so as to realize pedestrian re-identification of the pedestrian to be identified according to the similarity.
In an embodiment of the present invention, the step of performing preliminary processing on the acquired pedestrian image to generate a to-be-identified pedestrian image after the preliminary processing includes the following steps: selecting a first pedestrian image which reaches a preset resolution threshold value and a preset illumination threshold value and has a pedestrian shielding rate smaller than a preset shielding rate threshold value from the pedestrian image; the first pedestrian image is scaled to a preset size, and a second pedestrian image is obtained; and preprocessing the second pedestrian image to obtain the image of the pedestrian to be identified.
In an embodiment of the present invention, performing feature inference on the to-be-identified pedestrian image to obtain a first vector feature of the to-be-identified pedestrian on the to-be-identified pedestrian image includes the following steps: carrying out feature extraction on the image of the pedestrian to be recognized to generate a global pedestrian feature corresponding to the pedestrian to be recognized; segmenting the global features of the pedestrians to generate local features of the pedestrians; performing neighborhood fusion on the pedestrian local features to generate neighborhood cooperative local features; and performing feature fusion splicing on the neighborhood cooperative local features and the pedestrian global features to generate the first vector features.
In an embodiment of the invention, a residual error network or a lightweight convolutional neural network is adopted to perform feature extraction on the pedestrian image to be identified.
In an embodiment of the present invention, the global pedestrian feature is equally or unequally segmented, and three, five, and seven segmentations are performed on the global pedestrian feature respectively to generate three pedestrian local features corresponding to the three segmentations, five pedestrian local features corresponding to the five segmentations, and seven pedestrian local features corresponding to the seven segmentations.
In an embodiment of the present invention, the calculating the similarity between the first vector feature and the second vector feature includes the following steps: calculating a distance between the first vector feature and the second vector feature; and calculating the similarity according to the distance.
In an embodiment of the present invention, the distance is a cosine distance between the first vector feature and the second vector feature; the cosine distance is calculated by the following formula:
wherein f is i ' represents the first vector feature; f. of j ' represents the second vector feature; dist (f) i ′，f j ') represents a cosine distance between said first vector feature and said second vector feature;
the calculation formula of the similarity is as follows:
Similarity i,j ＝1-dist(f i ′，f j ′)；
wherein, the Similarity i，j Representing a similarity between the first vector feature and the second vector feature;
and when the similarity is greater than a preset threshold value, the pedestrian to be recognized and the target pedestrian are considered to be the same person.
The invention provides a pedestrian re-identification system based on neighborhood cooperative attention, which comprises: the device comprises a generating module, a first obtaining module, a second obtaining module and a calculating module; the generation module is used for carrying out primary processing on the obtained pedestrian image and generating a pedestrian image to be identified after the primary processing; the first acquisition module is used for performing feature inference on the image of the pedestrian to be identified to acquire a first vector feature of the pedestrian to be identified on the image of the pedestrian to be identified; the second acquisition module is used for carrying out feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image; the calculation module is used for calculating the similarity between the first vector feature and the second vector feature so as to realize pedestrian re-identification of the pedestrian to be identified according to the similarity.
The present invention provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described pedestrian re-identification method based on neighborhood cooperative attention.
The present invention provides a terminal, including: a processor and a memory; the memory is used for storing a computer program; the processor is used for executing the computer program stored in the memory so as to enable the terminal to execute the pedestrian re-identification method based on the neighborhood cooperative attention.
As described above, the pedestrian re-identification method, system, medium and terminal based on neighborhood collaborative attention according to the present invention have the following beneficial effects:
(1) compared with the prior art, the pedestrian re-identification method based on the neighborhood collaborative attention mechanism is adopted, the overall characteristics and the characteristics of the local area of the image are integrated, the neighborhood local characteristics of different scales are combined through the context neighborhood connection of the local characteristics, the local detail characteristics of different scales of pedestrians are better noticed, the accurate matching of the images of the same pedestrian is realized, and the accuracy of pedestrian re-identification is improved;
(2) the problem of partial shielding of pedestrians is better solved by performing the synergistic effect of neighborhood receptive fields on the local features, when the pedestrians are identified again, the pedestrians are focused on the common visible components, and the accuracy of pedestrian identification under dynamic environment imaging is remarkably improved;
(3) the method is suitable for quickly and effectively finding the specific pedestrian under the conditions that the camera equipment has slight errors, shooting scene changes, the pedestrian can be shielded by other objects to cause that a whole body image cannot be generated, and the like, and has flexible applicability.
Drawings
Fig. 1 is a flowchart illustrating a pedestrian re-identification method based on neighborhood cooperative attention according to an embodiment of the present invention.
FIG. 2 is a flow chart illustrating the generation of an image of a pedestrian to be identified according to an embodiment of the present invention.
FIG. 3 is a flow chart illustrating a feature of the present invention in obtaining a first vector.
FIG. 4 is a flow chart illustrating the calculation of similarity according to an embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a pedestrian re-identification model according to an embodiment of the invention.
FIG. 6 is a flow chart illustrating the operation of the feature inference module of the present invention in one embodiment.
FIG. 7 is a flowchart illustrating an embodiment of the present invention for obtaining neighborhood synergistic local features from global features of pedestrians.
Fig. 8 is a schematic structural diagram illustrating a pedestrian re-identification system based on neighborhood synergistic attention according to an embodiment of the present invention.
Fig. 9 is a schematic structural diagram of a terminal according to an embodiment of the invention.
Fig. 10 is a schematic structural diagram illustrating a segmentation from global features to local features in the prior art of pedestrian re-identification.
Description of the reference symbols
51 feature extraction module
52 feature inference module
53 feature matching module
81 generation module
82 first acquisition module
83 second acquisition module
84 calculation module
91 processor
92 memory
S1-S4
S11-S13
S21-S24 steps
S41-S42
Detailed Description
The following description of the embodiments of the present invention is provided by way of specific examples, and other advantages and effects of the present invention will be readily apparent to those skilled in the art from the disclosure herein. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the drawings only show the components related to the present invention rather than the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
Compared with the prior art, the pedestrian re-identification method, the system, the medium and the terminal based on the neighborhood collaborative attention system have the advantages that the pedestrian re-identification method based on the neighborhood collaborative attention system is adopted, the overall characteristics and the characteristics of the local area of the image are integrated, the context neighborhood connection of the local characteristics is utilized to combine the neighborhood local characteristics with different scales, the local detail characteristics with different scales of pedestrians are better noticed, the accurate matching of the images of the same pedestrian is realized, and the accuracy of pedestrian re-identification is improved; the problem of partial shielding of pedestrians is better solved by performing the synergistic effect of neighborhood receptive fields on the local features, when the pedestrians are identified again, the pedestrians focus on the common visible components, and the accuracy of pedestrian identification under dynamic environment imaging is remarkably improved; the method is suitable for quickly and effectively finding the specific pedestrian under the conditions that the camera equipment has slight errors, shooting scene changes, the pedestrian can be shielded by other objects, the whole body image cannot be generated and the like, and has flexible applicability.
As shown in fig. 1, in an embodiment, the method for re-identifying a pedestrian based on neighborhood synergistic attention of the present invention includes the following steps:
and step S1, carrying out primary processing on the acquired pedestrian image to generate a pedestrian image to be recognized after the primary processing.
It should be noted that, the pedestrian image is collected by the camera device (such as a snapshot machine), and because the pedestrian image collected by the camera device has the problems of different illumination, resolution, pedestrian ascending blocking rate in the image, and the like, when the pedestrian image is subsequently identified and judged, the pedestrian image meeting the requirements is firstly subjected to preliminary processing, and is screened out to be used as the pedestrian image to be identified, so that the efficiency of subsequent pedestrian re-identification is further improved.
As shown in fig. 2, in an embodiment, the preliminary processing is performed on the acquired pedestrian image, and the generating of the preliminary processed to-be-identified pedestrian image includes the following steps:
and S11, selecting a first pedestrian image which reaches a preset resolution threshold value and a preset illumination threshold value and has a pedestrian blocking rate smaller than a preset blocking rate threshold value from the pedestrian image.
Specifically, after the pedestrian image is obtained, the resolution, the illumination and the blocking rate of the pedestrian on the pedestrian image are respectively compared with a preset resolution threshold, a preset illumination threshold and a preset blocking rate threshold, and if the resolution is smaller than the preset resolution threshold or the illumination is smaller than the preset illumination threshold and the blocking rate of the pedestrian is larger than the preset blocking rate threshold, the pedestrian image is rejected, namely the pedestrian image is not used as the pedestrian image to be identified for subsequent pedestrian re-identification operation.
It should be noted that the preset resolution threshold, the preset illumination threshold, and the preset occlusion rate threshold are all preset, and the specific values are not used as conditions for limiting the present invention and can be determined according to the actual application scenario.
And step S12, the first pedestrian image is zoomed to a preset size, and a second pedestrian image is obtained.
Specifically, according to a preset size set in advance, after the first pedestrian image is acquired in step S11, the first pedestrian image is scaled to a preset size (such as 384 × 192) to acquire a scaled second pedestrian image.
And step S13, preprocessing the second pedestrian image to acquire the pedestrian image to be recognized.
It should be noted that the preprocessing operation includes, but is not limited to, a process of subtracting the pixel mean value from the second pedestrian image, and normalizing to the [ -1, 1] interval.
Further, after the processing in steps S11 to S13, the image of the pedestrian to be recognized is cut, so that the finally obtained image of the pedestrian to be recognized contains only one pedestrian to be recognized.
And step S2, performing feature inference on the to-be-identified pedestrian image to acquire a first vector feature of the to-be-identified pedestrian on the to-be-identified pedestrian image.
As shown in fig. 3, in an embodiment, performing feature inference on the image of the pedestrian to be recognized to obtain a first vector feature of the pedestrian to be recognized on the image of the pedestrian to be recognized includes the following steps:
and step S21, performing feature extraction on the image of the pedestrian to be recognized, and generating a global pedestrian feature corresponding to the pedestrian to be recognized.
In an embodiment, a Residual Network (ResNet) or a lightweight convolutional neural Network (MobileNet) is used to perform feature extraction on the pedestrian image to be identified.
It should be noted that the operation of extracting the features of the pedestrian image to be recognized is not limited to the use of a residual error network or a lightweight convolutional neural network; the residual network is commonly ResNet-50 and ResNet-101(50 and 101 respectively represent the number of hidden layers in ResNet); the residual error network and the lightweight convolutional neural network are common network structures in the technical field, so the detailed description of the specific structures of the residual error network and the lightweight convolutional neural network is omitted; in practical application, which network is specifically used for feature extraction of the image of the pedestrian to be identified can be determined according to practical situations.
Further, when the residual error network or the lightweight convolutional neural network is used for extracting the features of the pedestrian image to be identified, the convolutional layers in the residual error network and the lightweight convolutional neural network can be replaced by deformable convolutions.
And step S22, segmenting the global pedestrian features to generate the local pedestrian features.
Specifically, the global pedestrian feature generated in step S21 is segmented by an adaptive posing layer (pooling layer) to obtain subdivided multi-granularity pedestrian local features.
In one embodiment, the global pedestrian feature is subjected to three-part segmentation, five-part segmentation and seven-part segmentation respectively to generate three parts of local pedestrian features corresponding to the three parts of segmentation, five parts of local pedestrian features corresponding to the five parts of segmentation and seven parts of local pedestrian features corresponding to the seven parts of segmentation.
Specifically, before segmenting the global pedestrian feature obtained in step S21, the global pedestrian feature is copied into three parts to obtain four parts of global pedestrian features, and then three parts of global pedestrian features are respectively segmented into three parts, five parts of global pedestrian features and seven parts of global pedestrian features, so as to generate three parts of local pedestrian features corresponding to the three parts of segmentation, five parts of local pedestrian features corresponding to the five parts of segmentation and seven parts of local pedestrian features corresponding to the seven parts of segmentation.
Further, when the global features of the pedestrians are segmented, the global features can be segmented equally or unequally.
And step S23, performing neighborhood fusion on the local pedestrian features to generate neighborhood cooperative local features.
Specifically, after the global features of the pedestrian are respectively segmented by three, five and seven parts in the step S22, neighborhood fusion operations are respectively performed on the three parts of local features of the pedestrian generated by the corresponding three parts of segmentation, the five parts of local features of the pedestrian generated by the corresponding five parts of segmentation and the seven parts of local features of the pedestrian generated by the corresponding seven parts of segmentation through a concat layer, that is, feature fusion is respectively performed on adjacent local features of the pedestrian generated by each part of segmentation to generate corresponding neighborhood synergistic local features, so as to cope with the multi-scale local feature difference between different pedestrians.
It should be noted that, by selecting an odd number of parts of segmentation means, performing three, five and seven parts of segmentation on the global features of the pedestrians, and performing fusion on the local features of the neighborhood pedestrians, the fused neighborhood cooperative local features can be ensured to have the characteristics of multiple granularities and different scales, and information redundancy is avoided; for example, if the global feature of the pedestrian is segmented by two or four parts at the same time, two parts of local features (respectively marked as 0 and 1) of the pedestrian generated after the two parts of segmentation are subjected to neighborhood fusion operation, and a neighborhood synergistic local feature (marked as 0+1) is obtained, and is the same as the global feature of the pedestrian obtained in step S21, which causes information redundancy with the global feature of the pedestrian; after four parts of the pedestrian local features (respectively marked as 0, 1, 2 and 3) generated after the four parts of segmentation are subjected to neighborhood fusion operation, three parts of neighborhood cooperative local features (respectively marked as 0+1, 1+2 and 2+3) are obtained, and two parts of the three parts of neighborhood cooperative local features respectively correspond to the two parts of pedestrian local features generated after the two parts of segmentation (0 +1 of the three parts of neighborhood cooperative local features generated after the four parts of segmentation is the same as 0 of the two parts of pedestrian local features generated after the two parts of segmentation, and 2+3 of the three parts of neighborhood cooperative local features generated after the four parts of segmentation is the same as 1 of the two parts of pedestrian local features generated after the two parts of segmentation), which can also cause information redundancy; moreover, if three and six parts of segmentation are simultaneously carried out on the pedestrian global feature, three parts of pedestrian local features (respectively marked as 0, 1 and 2) generated after the three parts of segmentation are subjected to neighborhood fusion operation to obtain two parts of neighborhood synergistic local features (respectively marked as 0+1 and 1+ 2); after six parts of segmented six parts of pedestrian local features (respectively marked as 0, 1, 2, 3, 4 and 5) are subjected to neighborhood fusion operation, five parts of neighborhood synergistic local features (respectively marked as 0+1, 1+2, 2+3, 3+4 and 4+5) are obtained, three parts of the five neighborhood synergistic local features respectively correspond to the three parts of pedestrian local features generated after the three parts of segmentation (0 +1 of the five neighborhood synergistic local features generated after the six parts of segmentation is the same as 0 of the three parts of pedestrian local features generated after the three parts of segmentation, 2+3 of the five neighborhood synergistic local features generated after the six parts of segmentation is the same as 1 of the three parts of pedestrian local features generated after the three parts of segmentation, 4+5 of the five neighborhood synergistic local features generated after the six parts of segmentation is the same as 2 of pedestrians in the three parts of pedestrian local features generated after the three parts of segmentation), this also results in information redundancy.
In summary, when the global features of the pedestrians are segmented, the selection of the segmentation number needs to consider to avoid the final generation of information redundancy between the neighborhood cooperative local features or between the neighborhood cooperative local features and the local features of the pedestrians; of course, in addition to the above-mentioned three, five and seven parts of segmentation of the global pedestrian features, three, four and five parts of segmentation or three, four, seven parts of segmentation or four, five and seven parts of segmentation of the global pedestrian features may be performed, which may be determined according to actual conditions.
Furthermore, in addition to the above-mentioned three, five, and seven parts of segmentation are respectively performed on the global features of the pedestrians at a time, more parts of segmentation can be simultaneously performed on the global features of the pedestrians, for example, nine parts of segmentation are simultaneously performed on the global features of the pedestrians, but it should be noted that, on one hand, the operation of segmenting more parts can prolong the identification time of re-identification of the pedestrians, and reduce the identification efficiency; on the other hand, more parts of segmentation exists, and the generated neighborhood cooperative local features have information redundancy with the pedestrian local features or the neighborhood cooperative local features generated after three, five or seven parts of segmentation; for example, after nine segmentations are performed on the global features of the pedestrian at the same time, the generated neighborhood cooperative local features have partial information redundancy with the local features of the pedestrian generated after the three segmentations.
And S24, performing feature fusion and splicing on the neighborhood cooperative local features and the global pedestrian features to generate the first vector features.
Specifically, a concat layer is adopted to perform feature fusion splicing on the neighborhood collaborative local feature obtained in the step S23 and the pedestrian global feature obtained in the step S21 to generate a first vector feature, which is marked as f i ′；
f i ′＝concat([f G ′,f′ L_1 ,…f′ L_i …,f′ L_N ])；
Wherein, f' G Representing global features of the pedestrian; f' L_i Respectively representing the ith neighborhood cooperative local feature, and taking the value of i from 1 to N.
And step S3, performing feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image.
Specifically, the operations in the same steps S21-S24 are carried out on the target pedestrian image, and the feature inference of the target pedestrian image is realized to obtain a second vector feature, denoted as f, of the target pedestrian on the target pedestrian image j ′。
It should be noted that, the step S3 and the step S2 do not have a certain sequential execution order, and the step S2 may be executed first, and then the step S3 is executed, or the step S3 may be executed first, and then the step S2 is executed; in addition, step S3 may also be performed before step S1.
It should be noted that, by performing the synergistic effect of the neighborhood receptive field (in the convolutional neural network, determining the area size of the input layer corresponding to an element in an output result of a certain layer, which is called receptive field, using a mathematical language that is a mapping of the receptive field corresponding to an element of an output result of a certain layer in the convolutional neural network to the input layer, and then interpreting a point on the characteristic diagram to correspond to an area on the input diagram), the problem of partially blocking the pedestrian is better solved.
And step S4, calculating the similarity between the first vector feature and the second vector feature so as to realize pedestrian re-identification of the pedestrian to be identified according to the similarity.
As shown in fig. 4, in one embodiment, the calculating the similarity between the first vector feature and the second vector feature comprises the following steps:
and step S41, calculating the distance between the first vector characteristic and the second vector characteristic.
In one embodiment, the distance is a cosine distance between the first vector feature and the second vector feature; the formula for calculating the cosine distance is as follows:
wherein f is i ' represents the first vector feature; f. of j ' represents the second vector feature; dist (f) i ′,f j ') represents a cosine distance between the first vector feature and the second vector feature.
It should be noted that, instead of calculating the cosine distance between the first vector feature and the second vector feature, calculating the euclidean distance between the first vector feature and the second vector feature may be performed.
And step S42, calculating the similarity according to the distance.
In one embodiment, the similarity is calculated by the following formula:
Similarity i，j ＝1-dist(f i ′,f j ′)；
wherein, Similarity i，j Representing a degree of similarity between the first and second vector features.
It should be noted that when the similarity is greater than a preset threshold, the pedestrian to be recognized and the target pedestrian are considered to be the same person; otherwise, when the similarity is smaller than or equal to the preset threshold value, the pedestrian to be recognized and the target pedestrian are considered not to be the same person.
It should be noted that the preset threshold is preset and can be set empirically, and the specific value is not a condition for limiting the present invention.
The pedestrian re-identification method based on neighborhood cooperative attention of the present invention is further explained by the following specific embodiments.
The pedestrian re-identification method based on neighborhood cooperative attention is particularly applied to a shopping mall passenger flow scene, and analysis of passenger flow data is achieved through identification of pedestrians in the shopping mall.
Firstly, before the pedestrian re-recognition, the construction and training of a pedestrian re-recognition model are carried out.
As shown in fig. 5, a pedestrian re-identification model composed of a feature extraction module 51, a feature inference module 52 and a feature matching module 53 is constructed, wherein the feature inference module 52 includes a feature segmentation unit (adaptive boosting layer) and a neighborhood cooperative attention unit.
The feature extraction module 51 corresponds to the step S21, and has the same operation principle as the step S21; the feature inference module 52 corresponds to the above steps S22 to S24 (feature segmentation unit corresponds to step S22; neighborhood collaborative attention unit corresponds to steps S23 to S24), and has the same operation principle as the steps S22 to S24; the feature matching module 53 corresponds to the steps S41 to S42, and has the same operation principle as the steps S41 to S42.
Collecting pedestrian images collected by a market snapshot machine, cleaning, labeling (labeling the same ID of the same pedestrian on the pedestrian images) and primarily processing (the same as the primary processing principle recorded in the step S1) the pedestrian images, and dividing the pedestrian images into a training set and a test set according to a certain preset proportion; then training the pedestrian re-identification model through a training set; wherein, a proper deep learning frame is selected to train the feature extraction module 51, and a supervised learning mode is adopted to train the neighborhood cooperative attention unit.
It should be noted that, in the process of training the neighborhood cooperative attention unit, any one of the following functions may be used as a loss function to perform iterative training, including: the cross entropy, the triplet loss and the center loss enable the distance in the same category ID to be reduced as much as possible, and the distance between different categories ID to be increased as much as possible, so that a better classification effect is achieved.
After the pedestrian re-recognition model is trained through the training set, the pedestrian re-recognition model is evaluated through the testing set, the model with the optimal testing result is selected, the model with high generalization capability is obtained and serves as the final pedestrian re-recognition model, and subsequent pedestrian re-recognition operation is carried out.
Further, an average precision ratio ap (average precision) and an average precision mean value mapp (mean average precision) are used as evaluation indexes of pedestrian similarity measurement on the two-row human image; the AP measures the quality of the learned model in each class, the mAP measures the quality of the learned model in all classes, and the mAP is the average value of all APs.
Specifically, for the pedestrian image (denoted as M), by calculating the distance between the pedestrian image and other pedestrian images, N pedestrian images most similar to the pedestrian image can be obtained, and knowing the accuracy rate of the pedestrian belonging to the pedestrian in the pedestrian image M on each of the N pedestrian images, the average accuracy rate is the average value of the accuracy rates of the pedestrians belonging to the M on the N pedestrian images, namely:
wherein, the image is N; precision is the Precision rate of the pedestrians belonging to the pedestrian in the pedestrian image M on each pedestrian image in the N pedestrian images; and sigma Precision is the sum of the accuracy rates of the pedestrians belonging to the pedestrian in the pedestrian image M on each pedestrian image in the N pedestrian images.
When more than one pedestrian is present in the pedestrian image C to be retrieved, the average accuracy mean value mAP of all the pedestrians needs to be calculated:
wherein AP is the average accuracy of the pedestrians, C is the number of the pedestrians, and k is the kth pedestrian.
As shown in fig. 6 and 7, in this embodiment, ResNet-50 is selected as a feature extraction module, the preliminarily processed image of the pedestrian to be identified is sent to a ResNet-50 network after the fourth layer is modified (last stride is changed to 1), so as to obtain a 2048-dimensional global feature of the pedestrian, the global feature of the pedestrian is segmented by 3, 5, and 7 parts of the global feature of the pedestrian through a self-adaptive posing layer, so as to obtain three groups of differently-segmented multi-granularity local features of the pedestrian (which respectively correspond to one group of 0, 1, 2, 3, 4, and 5 and one group of 0, 1, 2, 3, 4, 5, and 6 in fig. 6), and the three groups of local features of the pedestrian are respectively subjected to neighborhood feature fusion processing, so as to obtain corresponding neighborhood synergistic local features (the specific neighborhood fusion principle is shown in fig. 7, which is to fuse adjacent local features generated after segmentation, and in fig. 7, the global feature is exemplified by the pedestrian in three parts, after three parts of global features of pedestrians are segmented, three parts of local features of the pedestrians are generated and are respectively represented by C1, C2 and C3, after neighborhood feature fusion, neighborhood cooperative local features of C1+ C2 obtained after C1 and C2 are fused and neighborhood cooperative local features of C2+ C3 obtained after C2 and C3 are fused are obtained, then all neighborhood cooperative local features and the global features of the pedestrians are fused and spliced, and a 4096-dimensional first vector feature is obtained; similarly, the operation is executed on the target pedestrian image to obtain a 4090-dimensional second vector feature; and (4) calculating the cosine distance and the similarity of the first vector feature and the second vector feature (the calculation mode is the same as that of the cosine distance and the similarity, and the description is not repeated here), and finally judging whether the pedestrian to be identified and the target pedestrian are the same person according to the similarity so as to realize pedestrian re-identification.
It should be noted that the protection scope of the pedestrian re-identification method based on neighborhood synergistic attention according to the present invention is not limited to the execution sequence of the steps listed in this embodiment, and all the solutions implemented by adding or subtracting steps and replacing steps in the prior art according to the principle of the present invention are included in the protection scope of the present invention.
As shown in fig. 8, in an embodiment, the system for pedestrian re-identification based on neighborhood synergistic attention of the present invention includes a generating module 81, a first obtaining module 82, a second obtaining module 83, and a calculating module 84.
The generation module 81 is used for performing preliminary processing on the acquired pedestrian image and generating a to-be-identified pedestrian image after the preliminary processing.
The first obtaining module 82 is configured to perform feature inference on the image of the pedestrian to be recognized, and obtain a first vector feature of the pedestrian to be recognized on the image of the pedestrian to be recognized.
The second obtaining module 83 is configured to perform feature inference on the target pedestrian image to obtain a second vector feature of the target pedestrian on the target pedestrian image.
The calculating module 84 is configured to calculate a similarity between the first vector feature and the second vector feature, so as to re-identify the pedestrian to be identified according to the similarity.
It should be noted that the structures and principles of the generating module 81, the first obtaining module 82, the second obtaining module 83, and the calculating module 84 correspond to the steps in the pedestrian re-identification method based on neighborhood cooperative attention one to one, and therefore, the details are not repeated here.
It should be noted that the division of the modules of the above system is only a logical division, and all or part of the actual implementation may be integrated into one physical entity or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the x module may be a processing element that is set up separately, or may be implemented by being integrated in a chip of the system, or may be stored in a memory of the system in the form of program code, and the function of the x module may be called and executed by a processing element of the system. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
The storage medium of the present invention stores thereon a computer program that, when executed by a processor, implements the above-described pedestrian re-identification method based on neighborhood cooperative attention. The storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic disk, U-disk, memory card, or optical disk.
As shown in fig. 9, the terminal of the present invention includes a processor 91 and a memory 92.
The memory 92 is used for storing computer programs; preferably, the memory 92 comprises: various media that can store program codes, such as ROM, RAM, magnetic disk, U-disk, memory card, or optical disk.
The processor 91 is connected to the memory 92, and is configured to execute the computer program stored in the memory 92, so that the terminal executes the above-mentioned pedestrian re-identification method based on neighborhood cooperative attention.
Preferably, the Processor 91 may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components.
It should be noted that the pedestrian re-identification system based on neighborhood synergistic attention of the present invention can implement the pedestrian re-identification method based on neighborhood synergistic attention of the present invention, but the implementation apparatus of the pedestrian re-identification method based on neighborhood synergistic attention of the present invention includes, but is not limited to, the structure of the pedestrian re-identification system based on neighborhood synergistic attention recited in the present embodiment, and all the structural modifications and substitutions of the prior art made according to the principle of the present invention are included in the protection scope of the present invention.
In summary, compared with the prior art, the pedestrian re-identification method, the system, the medium and the terminal based on the neighborhood cooperative attention of the invention integrate the overall characteristics of the image and the characteristics of the local area by adopting the pedestrian re-identification method based on the neighborhood cooperative attention mechanism, combine the neighborhood local characteristics of different scales through the context neighborhood connection of the local characteristics, better notice the local detail characteristics of different scales of the pedestrian, realize the accurate matching of the images of the same pedestrian, and improve the accuracy of pedestrian re-identification; the problem of partial shielding of pedestrians is better solved by performing the synergistic effect of neighborhood receptive fields on the local features, when the pedestrians are identified again, the pedestrians are focused on the common visible components, and the accuracy of pedestrian identification under dynamic environment imaging is remarkably improved; the method is suitable for quickly and effectively finding out the specific pedestrian under the conditions that the camera equipment has slight errors, shooting scene changes, the pedestrian can be shielded by other objects to cause that a whole body image cannot be generated, and the like, and has flexible applicability; therefore, the invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which may be made by those skilled in the art without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims (8)
1. A pedestrian re-identification method based on neighborhood synergistic attention is characterized by comprising the following steps:
performing primary processing on the obtained pedestrian image to generate a pedestrian image to be identified after the primary processing;
performing feature inference on the pedestrian image to be identified to acquire a first vector feature of the pedestrian to be identified on the pedestrian image to be identified; the characteristic reasoning is carried out on the image of the pedestrian to be recognized, and the step of obtaining the first vector characteristic of the pedestrian to be recognized on the image of the pedestrian to be recognized comprises the following steps:
performing feature extraction on the image of the pedestrian to be identified to generate a global pedestrian feature corresponding to the pedestrian to be identified;
segmenting the global features of the pedestrians to generate local features of the pedestrians; equally or unequally dividing the global pedestrian feature, and respectively carrying out three-part division, five-part division and seven-part division on the global pedestrian feature to generate three parts of local pedestrian features corresponding to the three parts of division, five parts of local pedestrian features corresponding to the five parts of division and seven parts of local pedestrian features corresponding to the seven parts of division;
performing neighborhood fusion on the pedestrian local features to generate neighborhood cooperative local features; the neighborhood fusion is to perform feature fusion on the local features of the pedestrians adjacent to each other;
performing feature fusion splicing on the neighborhood cooperative local features and the pedestrian global features to generate the first vector feature;
performing feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image;
and calculating the similarity between the first vector feature and the second vector feature so as to realize pedestrian re-identification of the pedestrian to be identified according to the similarity.
2. The pedestrian re-identification method based on neighborhood cooperative attention according to claim 1, wherein the step of performing preliminary processing on the obtained pedestrian image to generate a preliminary processed pedestrian image to be identified comprises the following steps:
selecting a first pedestrian image which reaches a preset resolution threshold value and a preset illumination threshold value and has a pedestrian shielding rate smaller than a preset shielding rate threshold value from the pedestrian image;
the first pedestrian image is scaled to a preset size, and a second pedestrian image is obtained;
and preprocessing the second pedestrian image to obtain the image of the pedestrian to be identified.
3. The pedestrian re-identification method based on neighborhood cooperative attention according to claim 1, wherein a residual error network or a lightweight convolutional neural network is adopted to perform feature extraction on the pedestrian image to be identified.
4. The pedestrian re-identification method based on neighborhood synergistic attention according to claim 1, wherein calculating the similarity between the first vector feature and the second vector feature comprises the steps of:
calculating a distance between the first vector feature and the second vector feature;
and calculating the similarity according to the distance.
5. The neighborhood synergistic attention based pedestrian re-identification method according to claim 4, wherein said distance is a cosine distance between said first vector feature and said second vector feature; the cosine distance is calculated by the following formula:
wherein f is i ' representing said first vector feature; f. of j ' represents the second vector feature; dist (f) i ′,f j ') represents a cosine distance between said first vector feature and said second vector feature;
the calculation formula of the similarity is as follows:
Similarity i,j ＝1-dist(f i ′,f j ′)；
wherein, the Similarity i,j Representing a similarity between the first vector feature and the second vector feature;
and when the similarity is greater than a preset threshold value, the pedestrian to be recognized and the target pedestrian are considered to be the same person.
6. A pedestrian re-identification system based on neighborhood synergistic attention, comprising: the device comprises a generating module, a first obtaining module, a second obtaining module and a calculating module;
the generation module is used for carrying out primary processing on the obtained pedestrian image and generating a pedestrian image to be identified after the primary processing;
the first acquisition module is used for performing feature inference on the image of the pedestrian to be identified to acquire a first vector feature of the pedestrian to be identified on the image of the pedestrian to be identified; the characteristic reasoning is carried out on the image of the pedestrian to be recognized, and the step of obtaining the first vector characteristic of the pedestrian to be recognized on the image of the pedestrian to be recognized comprises the following steps:
performing feature extraction on the image of the pedestrian to be identified to generate a global pedestrian feature corresponding to the pedestrian to be identified;
segmenting the global features of the pedestrians to generate local features of the pedestrians; equally dividing or unequally dividing the pedestrian global feature, and respectively performing three-part division, five-part division and seven-part division on the pedestrian global feature to generate three parts of pedestrian local features corresponding to the three-part division, five parts of pedestrian local features corresponding to the five-part division and seven parts of pedestrian local features corresponding to the seven-part division;
performing neighborhood fusion on the pedestrian local features to generate neighborhood cooperative local features; the neighborhood fusion is to perform feature fusion on the local features of the adjacent pedestrians;
performing feature fusion splicing on the neighborhood cooperative local features and the pedestrian global features to generate the first vector feature;
the second acquisition module is used for carrying out feature inference on the target pedestrian image to acquire a second vector feature of the target pedestrian on the target pedestrian image;
the calculation module is used for calculating the similarity between the first vector feature and the second vector feature so as to realize the pedestrian re-identification of the pedestrian to be identified according to the similarity.
7. A storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the neighborhood collaborative attention based pedestrian re-identification method of any one of claims 1 to 5.
8. A terminal, comprising: a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to execute the memory-stored computer program to cause the terminal to perform the neighborhood collaborative attention based pedestrian re-identification method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010772245.1A CN112036250B (en) | 2020-08-04 | 2020-08-04 | Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010772245.1A CN112036250B (en) | 2020-08-04 | 2020-08-04 | Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112036250A CN112036250A (en) | 2020-12-04 |
| CN112036250B true CN112036250B (en) | 2022-09-09 |
Family
ID=73583813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010772245.1A Active CN112036250B (en) | 2020-08-04 | 2020-08-04 | Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112036250B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112861691B (en) * | 2021-01-29 | 2022-09-09 | 中国科学技术大学 | Pedestrian re-identification method under occlusion scene based on part perception modeling |
| CN113205177B (en) * | 2021-04-25 | 2022-03-25 | 广西大学 | Electric power terminal identification method based on incremental collaborative attention mobile convolution |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109657533A (en) * | 2018-10-27 | 2019-04-19 | 深圳市华尊科技股份有限公司 | Pedestrian recognition methods and Related product again |
| CN109857889A (en) * | 2018-12-19 | 2019-06-07 | 苏州科达科技股份有限公司 | A kind of image search method, device, equipment and readable storage medium storing program for executing |
| CN110348282A (en) * | 2019-04-30 | 2019-10-18 | 贵州大学 | The method and apparatus identified again for pedestrian |
-
2020
- 2020-08-04 CN CN202010772245.1A patent/CN112036250B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109657533A (en) * | 2018-10-27 | 2019-04-19 | 深圳市华尊科技股份有限公司 | Pedestrian recognition methods and Related product again |
| CN109857889A (en) * | 2018-12-19 | 2019-06-07 | 苏州科达科技股份有限公司 | A kind of image search method, device, equipment and readable storage medium storing program for executing |
| CN110348282A (en) * | 2019-04-30 | 2019-10-18 | 贵州大学 | The method and apparatus identified again for pedestrian |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112036250A (en) | 2020-12-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sindagi et al. | Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting | |
| Shami et al. | People counting in dense crowd images using sparse head detections | |
| Wang et al. | Static and moving object detection using flux tensor with split Gaussian models | |
| Zand et al. | Multiscale crowd counting and localization by multitask point supervision | |
| Khan et al. | Revisiting crowd counting: State-of-the-art, trends, and future perspectives | |
| CN111104925B (en) | Image processing method, image processing apparatus, storage medium, and electronic device | |
| CN109902576B (en) | Training method and application of head and shoulder image classifier | |
| Zhai et al. | Deep texton-coherence network for camouflaged object detection | |
| CN112036250B (en) | Pedestrian re-identification method, system, medium and terminal based on neighborhood cooperative attention | |
| Rajasegaran et al. | On the benefits of 3d pose and tracking for human action recognition | |
| Cho et al. | Semantic segmentation with low light images by modified CycleGAN-based image enhancement | |
| CN111259919A (en) | Video classification method, device and equipment and storage medium | |
| Sahu et al. | Single image dehazing via fusion of multilevel attention network for vision-based measurement applications | |
| CN115239765B (en) | Infrared image target tracking system and method based on multi-scale deformable attention | |
| Haggui et al. | Human detection in moving fisheye camera using an improved YOLOv3 framework | |
| Pais et al. | Omnidrl: Robust pedestrian detection using deep reinforcement learning on omnidirectional cameras | |
| Dai et al. | Oamatcher: An overlapping areas-based network for accurate local feature matching | |
| Zhang et al. | CE-RetinaNet: A channel enhancement method for infrared wildlife detection in UAV images | |
| CN114612545A (en) | Image analysis method and training method, device, equipment and medium of related model | |
| Ammous et al. | Improved YOLOv3-tiny for silhouette detection using regularisation techniques. | |
| Nguyen et al. | SmithNet: strictness on motion-texture coherence for anomaly detection | |
| Abdullah et al. | Context aware crowd tracking and anomaly detection via deep learning and social force model | |
| Mucha et al. | Depth and thermal images in face detection-a detailed comparison between image modalities | |
| Kizrak et al. | Crowd density estimation by using attention based capsule network and multi-column CNN | |
| Jiashu | Performance analysis of facial recognition: A critical review through glass factor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 201203 No. 6, Lane 55, Chuanhe Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai Applicant after: Winner Technology Co.,Ltd. Address before: 201505 Room 216, 333 Tingfeng Highway, Tinglin Town, Jinshan District, Shanghai Applicant before: Winner Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |