CN117173794A - Pedestrian re-identification method suitable for edge equipment deployment - Google Patents
Pedestrian re-identification method suitable for edge equipment deployment Download PDFInfo
- Publication number
- CN117173794A CN117173794A CN202311450373.4A CN202311450373A CN117173794A CN 117173794 A CN117173794 A CN 117173794A CN 202311450373 A CN202311450373 A CN 202311450373A CN 117173794 A CN117173794 A CN 117173794A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- frame
- target
- model
- filtering
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000001914 filtration Methods 0.000 claims abstract description 71
- 238000001514 detection method Methods 0.000 claims description 61
- 238000004364 calculation method Methods 0.000 claims description 16
- 230000004927 fusion Effects 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 16
- 238000010606 normalization Methods 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 238000011156 evaluation Methods 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 9
- 101150053100 cls1 gene Proteins 0.000 claims description 5
- 230000033001 locomotion Effects 0.000 claims description 5
- 238000013139 quantization Methods 0.000 claims description 4
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 3
- 238000013145 classification model Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 claims description 3
- 230000005764 inhibitory process Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 101150115304 cls-2 gene Proteins 0.000 claims description 2
- 238000012805 post-processing Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 238000000605 extraction Methods 0.000 description 9
- 238000013441 quality evaluation Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The application discloses a pedestrian re-identification method suitable for edge equipment deployment, which comprises the following steps: obtaining a pedestrian envelope frame on each frame of image, and endowing an id for each pedestrian envelope frame and tracking; filtering the pedestrian envelope frames with the same id; extracting pedestrian features and pedestrian attributes, fusing the pedestrian features and the pedestrian attributes of the same id, importing the fused pedestrian features into a database, adopting mean shift clustering in the database, further fusing the pedestrian features among different ids, removing redundant ids, and finally obtaining pedestrian features and attribute information of different ids; and finally, re-identifying pedestrians. The pedestrian re-identification method provided by the application reduces the calculated amount, ensures the accuracy of the final inquiry effect of the scheme, and is more suitable for the deployment of edge equipment.
Description
Technical Field
The application relates to the field of edge calculation, in particular to a pedestrian re-identification method suitable for edge equipment deployment.
Background
The pedestrian re-recognition technology is a research hotspot in the field of computer vision, can realize cross-border search of the same pedestrian under different monitoring cameras, can be used for lost person search, pedestrian track generation and active area positioning, and has wide application prospects in market and park management.
The existing pedestrian re-identification method based on deep learning mainly utilizes fusion of multiple information such as face information, pedestrian attribute information, pedestrian global or local abstract features and the like to compare pedestrians, and the pedestrian re-identification method is high in similarity, namely the same person. For a pedestrian re-identification system, the number of the processed monitoring video is large, the generated data is huge, and if the huge data is directly transmitted to a central server, a large amount of bandwidth is occupied and a large amount of computation load is consumed. With the advent of the worldwide interconnecting age, edge computing devices are widely used to replace part or all of the central server tasks to relieve the bandwidth pressure and computing energy consumption of the servers.
The technical problem of the method is that the traditional multi-information fusion of pedestrian re-identification is realized, a plurality of algorithm models and more complex business logic are needed to cooperate, the calculation power of the main stream edge equipment is limited at present, even if the main stream edge equipment is realized, the delay is higher, or the proposed scheme is difficult to achieve the balance of precision and efficiency on the edge equipment.
For example, the application number is 201910099228.3, the application is a self-adaptive city searching method based on edge calculation, and the calculated amount is reduced by filtering redundant video frames; pedestrian detection is carried out on the uploaded video frames, pedestrian re-identification is carried out on a certain proportion of detected pedestrians, and characteristics of the pedestrians are output; the edge server uploads K pedestrians with the highest identified and found target similarity to the cloud server at regular time; and re-identifying pedestrians in the remaining proportion by utilizing a pedestrian re-identification module on the cloud server, combining identification results of the cloud server and the edge server, and returning the identified K pedestrians with the highest similarity with the search target to the terminal at fixed time for processing. The technical scheme has the following problems: (1) The model extracts single pedestrian characteristic information, and more interference exists in practical application; (2) Filtering redundant video frames mainly utilizes the similarity of image histograms as global features, and easily ignoring the difference of local features, so that effective information is lost; (3) The task of extracting features is proportionally distributed to the server and the side devices, and if the calculation force or bandwidth difference between the server and the side devices is large, the synchronization problem still occurs.
For another example, the application number is 202211708842.3, and the application is the technical scheme of the pedestrian re-identification method, system and device of the distributed edge collaborative reasoning, and the image to be processed is obtained through the distributed edge collaborative reasoning architecture; extracting identity features and attribute features of an image through ResNet-50 to obtain a first classification result of the identity of a pedestrian in the image; obtaining attribute classification results of all pedestrians in the image according to the attribute characteristics; carrying out attribute feature fusion according to the attribute classification result to obtain fusion attribute features, and obtaining a second classification result of the identity of the pedestrian in the image according to the fusion attribute features; and carrying out pedestrian re-recognition according to the first classification result and the second classification result. The technical scheme has the following problems: (1) The Resnet50 is adopted to extract the identity characteristics and the attribute characteristics of the image, the model is large, and the edge end reasoning consumes long time; (2) No filtering mechanism is proposed and the full video sequence processing is computationally intensive.
For another example, the application number is CN202211138091.6, and the application is named as a pedestrian re-recognition and tracking method based on edge calculation, firstly, multi-angle feature extraction is performed on pedestrians, single-camera feature detection is performed, and a multi-angle initial feature set of the pedestrians is constructed; then acquiring pedestrian characteristics and tracking target pedestrians; finally, searching pedestrian characteristics, and re-identifying pedestrians to obtain target pedestrian data; the multi-angle pedestrian characteristics are collected at the scene entrance and the scene exit, and the pedestrian re-recognition and pedestrian tracking algorithm is fused to accelerate calculation and be deployed at the edge side, so that the re-recognition and tracking result with good real-time performance can be provided. The technical scheme has the following problems: (1) The scheme has limitation, and authors limit both the initial feature set and the acquisition place, so that the scheme is not applicable to wider scenes; (2) The Fairmot is used for simultaneously carrying out pedestrian tracking and pedestrian feature extraction, the Fairmot is a multi-target detection and tracking algorithm, the capability of acquiring local features is weak, and the pedestrian features extracted by the Fairmot are directly used for pedestrian re-recognition, so that false recognition can exist.
Disclosure of Invention
The application aims to overcome the defects and shortcomings of the prior art and provide a pedestrian re-identification method suitable for edge equipment deployment, which starts from the positions of model improvement and business logic and reduces the calculated amount under the condition of ensuring the precision.
The aim of the application is achieved by the following technical scheme:
a pedestrian re-identification method suitable for edge equipment deployment comprises the following steps:
s1, obtaining pedestrian envelope frames on each frame of image through a target detection model, endowing each pedestrian envelope frame with an id by utilizing a sort multi-target tracking model, and tracking all the pedestrian envelope frames; filtering the pedestrian envelope frames with the same id;
s2, simultaneously extracting pedestrian features and pedestrian attributes from the pedestrian images through a fastreid model, fusing multi-frame pedestrian features and pedestrian attributes of the same id, importing the fused pedestrian features into a database, adopting mean shift clustering in the database, further fusing the pedestrian features among different ids, removing redundant ids, and finally obtaining pedestrian features and attribute information of different ids;
s3, performing cosine distance calculation on pedestrian features of different ids to obtain K similar pedestrians, and then performing secondary filtering by using pedestrian attribute information of the K similar pedestrians to filter attribute mismatch results in the K similar pedestrians so as to obtain the most similar pedestrian re-recognition result.
In step S1, the target detection model is a yolox_s model after light weight.
The lightweight process of the yolox_s model comprises the following steps:
(1) Performing batch norm weight sparse training on the yolox_s model, and then cutting a channel with the bn layer weight of the yolox_s model smaller than a preset value;
(2) Performing finetune fine tuning training on the trimmed yolox_s model;
(3) The yolox_s model performs int8 quantization when the edge device is migrated.
The bn layer of the yolox_s model is a feature normalization operator of the yolox_s model, the feature normalization operator is a batch norm2d operator, and the formula is as follows:
；
wherein,for input data requiring normalization, +.>And->For the mean and variance of the batch data, +.>Variable added to prevent zero occurrence of denominator, < >>And->Affine operation, namely linear transformation is carried out on the input value;from the formula, the weight of bn layer is +.>And->Output->Each channel in (a) has a corresponding +.>And->Use +.>Judging the importance degree of different channels; when->If the weight of the channel is smaller than the preset value, the weight of the channel is cut if the follow-up influence of the data of the channel on the model is smaller; the goal of the catch-norm weight sparse training procedure is to obtain a fraction of smaller +.>The value of the whole channel cut-out ratio is 40%, namely, the part with 40% smaller is +.>And clipping the channel corresponding to the value.
In step S1, the working process of the target detection model is as follows:
firstly, acquiring image data from a camera terminal, preprocessing the image data, and inputting the preprocessed image data into a target detection model yolox_s for reasoning: the image data is subjected to convolution operation on a backbone network of a target detection model yolox_s to obtain an output characteristic diagram; then inputting the feature map to an output head of a target detection model yolox_s to carry out regression and classification of a target frame;
the output head of the object detection model yolox_s has three branches in total: cls_output is used for predicting the class and the fraction of the target frame, the output size is W.H.C, wherein W is the width of the output feature map, H is the height of the output feature map, and C is the number of detection classes; the obj_output is used for judging whether the target frame is a foreground or a background, and the output size is W.times.H.times.1; reg_output used for predicting coordinate information (x, y, W, H) of the target frame, and outputting a W x 4;
combining three branches of an output head of a target detection model yolox_s to obtain final candidate prediction frame information, wherein the size of the final candidate prediction frame information is pred_num_dim, wherein pred_num=w_h, the number of the prediction frames is represented, dim=c+1+4, and the feature vector dimension of each prediction frame is represented; each prediction frame contains a feature vector with a dimension dim, [ cls_1 cls_2 ]. Cls_C obj x y h ], wherein x is x coordinate information of a center point of the target frame, y is y coordinate information of the center point of the target frame, w is width information of the target frame, h is height information of the target frame, obj is score information of the target frame, cls_1 is score of a category 1 of the target frame, and cls_C is score of a category C of the target frame;
post-processing candidate prediction frame results: firstly, reducing the normalized coordinates of the predicted frame to the actual pixel size, then filtering the redundant predicted frame by utilizing non-maximum value inhibition, and outputting to obtain the final pedestrian envelope frame.
The preprocessing includes normalized scaling of image size and normalization of data.
In step S1, the working process of the sort multi-target tracking model is as follows:
(1) An id is assigned to each pedestrian envelope frame obtained by the target detection model;
(2) Initializing a target tracker: for each detected pedestrian target, creating a Kalman filtering target tracker for each detected pedestrian target, wherein the Kalman filtering target tracker is used for tracking the motion of the pedestrian target in the subsequent image frames;
(3) Multi-frame tracking: for each frame, the following operations are performed:
1) And (3) data association: matching a pedestrian envelope frame bboxmeasurement obtained by a Kalman filtering prediction pedestrian envelope frame bboxprediction and a target detection model by using a Hungary algorithm;
2) And (5) updating the state: and updating the current state of a pedestrian envelope frame bboxmeasurement obtained by using a Kalman filtering prediction pedestrian envelope frame bboxprediction and a target detection model to obtain a pedestrian tracking envelope frame bboxoptimal as a tracking result.
In the step S1, the filtering comprises space-time filtering, quality filtering and posture assessment which are sequentially carried out; the space-time filtering means that for a certain pedestrian tracking result with existing id, only the pedestrian tracking result with the frame interval larger than the time threshold and the space interval larger than the space threshold of the previous frame image is reserved, and then the reserved pedestrian tracking result enters the quality filtering; the quality filtering means filtering the pedestrian tracking results which do not meet the quality index, and scoring each quality index of the pedestrian tracking results at the same time, wherein the weighted summation of all the quality index scores is the quality score S of the pedestrian tracking results 1 Then, the reserved pedestrian tracking result enters into gesture evaluation; the gesture evaluation is to divide the pedestrian gesture into a plurality of attributes, and output probability values of the plurality of pedestrian gestures and quality scores S through an attribute classification model 1 Weighted summation to obtain final pedestrian quality score S final ；
Inputting a pedestrian tracking result through space-time filtering, quality filtering and attitude evaluation: if the corresponding id in the pedestrian tracking result is stored with n pedestrian tracking results, if n is smaller than the frame number corresponding to the id, directly inputting, if n is greater than or equal to the frame number corresponding to the id, comparing the pedestrian quality scores S final Pedestrian quality score S if pedestrian tracking result final And if the score is larger than the minimum value of the scores of the n pedestrian tracking results, replacing.
In step S2, the pedestrian features and the pedestrian attributes are extracted from the pedestrian image through the fastreid model at the same time, which specifically includes the following steps: firstly, intercepting pedestrians in an image through a reserved pedestrian envelope frame, and then, preprocessing to obtain standard data; and then importing the standard data into a fastreid model, obtaining pedestrian characteristics after convolution operation, and outputting pedestrian attribute classification results after the pedestrian characteristics pass through a linear classifier.
In step S2, the multi-frame pedestrian feature fusion is obtained by a weighted summation mode; the multi-frame pedestrian attribute adopts a high-low double-threshold mode to judge the attribute, and performs result statistics on each attribute, and the state with the highest accumulated value is the pedestrian attribute fusion result; by fusing, one id corresponds to only one pedestrian feature result and one pedestrian attribute result.
Meanwhile, the application provides:
a server comprising a processor and a memory, wherein at least one section of program is stored in the memory, and the program is loaded and executed by the processor to realize the pedestrian re-identification method suitable for edge device deployment.
A computer readable storage medium having stored therein at least one program loaded and executed by a processor to implement the pedestrian re-identification method described above as being suitable for edge device deployment.
Compared with the prior art, the application has the following advantages and beneficial effects:
1. for quality evaluation and filtering of pedestrian pictures, multiple filtering (space-time, quality and gesture) and multiple quality evaluation indexes (resolution, definition, integrity, brightness and aspect ratio) are adopted to evaluate and filter the pedestrian pictures, so that on one hand, distortion pictures affecting accuracy are removed, on the other hand, redundant pictures affecting calculation efficiency, including spatial redundancy, temporal redundancy and gesture redundancy, are filtered, and the gesture is utilized as a scoring reference, so that recognition influence caused by loss of side gesture information is avoided. Compared with the prior art, the redundancy of the whole frame of image is not directly and globally judged, and the loss of local information is avoided.
2. In order to adapt to edge end deployment, the application adopts a multitask model to extract pedestrian characteristics and pedestrian attributes, has multiple functions, and effectively reduces the calculated amount while ensuring the richer information extraction capability. In addition, the model is subjected to light weight processing, so that the model reasoning speed is further improved. Compared with the prior art, the model in the prior art is unreasonable in design, is not improved in light weight, is not suitable for edge end deployment, is single in feature extraction result, and is not strong in extraction capability.
3. The application fuses the information in the same id, including the fusion of the characteristics and the attributes, on one hand, the time-space information characteristics are obtained, and on the other hand, the calculated data volume is reduced; for different ids, clustering fusion is also performed, and redundancy caused by pedestrian tracking errors among ids is reduced. The prior art mainly fuses the characteristics, but does not see the condition of fusing the attributes.
4. The query process of the application carries out secondary judgment, and the pedestrian query process comprises two flows: and the feature comparison and the attribute comparison are carried out to search and filter similar pedestrians from the angles of the global features and the local features, so that the accuracy of the query result is improved.
Drawings
Fig. 1 is a flowchart of a pedestrian feature acquisition phase according to the present application.
Fig. 2 is a flowchart of the pedestrian re-recognition phase according to the present application.
Detailed Description
The present application will be described in further detail with reference to examples and drawings, but embodiments of the present application are not limited thereto.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
A pedestrian re-recognition method suitable for edge equipment deployment comprises two stages, namely a pedestrian characteristic acquisition stage and a pedestrian re-recognition stage.
1. Pedestrian feature acquisition phase (see FIG. 1)
1.1 pedestrian target detection
The adopted target detection algorithm is yolox_s, yolox adopts yolov3_spp as a reference model, and a decoupling Head, namely a coupled Head, an Anchor-free and other tricks are adopted for improvement, so that the Yolox-Darknet53 network structure is finally obtained.
The pedestrian target detection algorithm flow is as follows: firstly, acquiring image data from a camera terminal, and preprocessing the image data, wherein the preprocessing comprises standardized scaling of image size and normalization processing of data; the preprocessed image data are input into a yolox_s model for reasoning, the model mainly comprises a main network and an output head, and the data are subjected to convolution operation of various operators such as a convolution layer, a normalization layer, an activation layer and the like in the main network to obtain an output characteristic diagram; the feature map is then input to the output head for regression and classification of the target box. The output head has three branches in total: cls_output, mainly predicting the class and the fraction of the target frame, wherein the output size is W.times.H.times.C; obj_output is used for mainly judging whether a target frame is a foreground or a background, and the output size is W.times.H.times.1; reg_output is used for mainly predicting coordinate information (x, y, W and H) of a target frame, and outputting a W x 4; wherein W is the width of the output feature map, H is the height of the output feature map, C is the number of detection categories, for example, when the detection categories are only pedestrians, C is 1; and finally, combining the three output to obtain final candidate prediction frame information, wherein the size of the final candidate prediction frame information is pred_num, the pred_num=w×h, the number of the prediction frames is represented, the dim=c+1+4, and the feature vector dimension of each prediction frame is represented. Each prediction box contains a feature vector with dimension dim, [ cls_1cls_2..cls_c obj x ywh ]; x is the x coordinate information of the center point of the target frame; y is the y coordinate information of the center point of the target frame; w, target frame width information; h, target frame height information; obj is target frame score information; cls_1, target frame class 1 score; cls_C, target box category C score. The candidate prediction frames result in a large number and require filtering. There is then a subsequent post-treatment process. And firstly reducing the normalized coordinates of the predicted frame to the actual pixel size, then filtering the redundant predicted frame by utilizing non-maximum value inhibition, and outputting to obtain the final pedestrian target detection envelope frame.
In order to improve the operation efficiency of the pedestrian target detection algorithm, the application also performs light weight processing on the yolox_s model, which comprises the following operationsThe method comprises the following steps: (1) Performing the sparse training of the weight of the batch norm layer, and then cutting the weight of the batch norm layerSmaller channels, overall channel cut-out ratio 40%; (2) performing finetune fine tuning training on the model after clipping; and (3) carrying out int8 quantization when the model is transplanted to the edge equipment. The pedestrian detection envelope frame can be obtained in real time by the target detection algorithm after the light weight processing.
The batch norm layer is a feature normalization operator in the network, this time is a batch norm2d operator, and the operator formula is as follows:
；
wherein,for input data requiring normalization, +.>And->For the mean and variance of the batch data, +.>Variable added to prevent zero occurrence of denominator, < >>And->The input values are subjected to affine operation, namely linear transformation, and the values are normalized information of the model on the training data of the current scene in the training process. From the formula, the weight of bn layer is +.>And->Transport and deliverGo out->Each channel in (a) has a corresponding +.>And->Use +.>Judging the importance degree of different channels; when->If the weight of the channel is smaller than the preset value, the weight of the channel is cut if the follow-up influence of the data of the channel on the model is smaller; the goal of the catch-norm weight sparse training procedure is to obtain a fraction of smaller +.>The value of the whole channel cut-out ratio is 40%, namely, the part with 40% smaller is +.>And clipping the channel corresponding to the value. The clipped pedestrian detection yolox_s model has smaller video memory occupation and obvious speed improvement.
The pedestrian detection envelope frame on each frame of image can be obtained by transmitting the image frames obtained by image acquisition to a pedestrian detection yolox_s model.
1.2 pedestrian target tracking
In order to improve the accuracy of pedestrian re-identification, the space-time information of pedestrians can be introduced, namely, the target detection result of the same continuous multi-frame pedestrians is obtained, and the process can be realized through a tracking algorithm, and the specific flow is as follows:
the pedestrian target detection envelope frame of the current image can be obtained through the pedestrian target detection algorithm of 1.1, each envelope frame is mainly endowed with an id through the tracking algorithm, the predicted envelope frame of each envelope frame in the next frame is predicted, the pedestrian target detection envelope frame of the next frame corresponds to the predicted envelope frame one by one, and finally a tracking envelope frame result is obtained through updating.
The application adopts a concise SORT multi-target tracking algorithm to endow id to pedestrian detection results and continuously track, and the detection results of the same id are regarded as the same pedestrian, and the working process is as follows:
(1) Pedestrian target detection: using a target detection algorithm of 1.1 to detect the pedestrian target of the input image frame, and endowing each envelope frame with a unique id;
(2) Initializing a target tracker: for each detected target, a Kalman filter target Tracker (Tracker) is created for it. Is responsible for tracking the motion of the object in subsequent image frames.
(3) Multi-frame tracking: for each frame, the following operations are performed:
1) And (3) data association: matching an envelope frame bboxmeasurement obtained by the Kalman filtering prediction envelope frame bboxprediction and a target detection algorithm by using a Hungary algorithm;
2) And (5) updating the state: and updating the current state by using the bboxprediction and the bboxmeasurement to obtain a tracking envelope frame bboxoptimal as a tracking result.
1.3 pedestrian image scoring and filtering
After the pedestrian target tracking is carried out, target detection results of the same pedestrian continuous multiframe are obtained, in the detection results, distortion conditions of too small resolution, motion blurring, shielding and too dark light exist, the detection results are directly used for subsequent feature extraction and comparison and can influence the accuracy of recognition, the detection results are required to be filtered, and the information-rich results are reserved. Meanwhile, in the whole life cycle of the monitoring video of the same pedestrian, the situations of space redundancy, time redundancy and gesture redundancy exist, that is, information redundancy does not exist only in the pedestrian detection results obtained in different spaces, different times and different gestures. By processing the distortion condition and the redundancy condition, on one hand, the subsequent recognition accuracy can be improved, on the other hand, the data volume is reduced, and the subsequent data processing pressure is reduced. The pedestrian image scoring and filtering scheme mainly comprises triple filtering, namely space-time filtering, quality filtering and gesture filtering. The filtering result is stored in the form of id sets, and pedestrian detection results under m discontinuous frames are stored in each id at most.
1.3.1 space-time filtration
For a certain image frame n, traversing all pedestrian tracking results, and for a certain pedestrian tracking result P i The position coordinates are (x i ,y i ) If the pedestrian detection result of the previous frame is never saved in the filtering results, the next round of filtering is directly entered. If the id exists, the pedestrian tracking result P of the last time is obtained i-1 Corresponding frame number m, coordinates (x i-1 ,y i-1 ) If the frame interval n-m is smaller than the time threshold T time Filtering the pedestrian tracking result P i . Similarly, if (x i ,y i ) And (x) i-1 ,y i-1 ) The coordinate distance is less than the spatial threshold T dist Filtering the pedestrian tracking result P i . And when the two thresholds meet the conditions, entering the next round of filtering process.
1.3.2 Mass filtration
P entering the filter i Quality assessment of several indices, resolution, sharpness, integrity, brightness and aspect ratio, respectively, was performed. Each index produces a score, and the weighted sum of all scores is the final quality score S 1 。
Resolution ratio: calculation of P i W and h, filtering the pedestrian tracking result P when w or h is less than the minimum resolution threshold Tsize i 。
Definition: calculation of P using Reblur secondary blurring sharpness algorithm i Compared with other definition algorithms based on gradients, the definition value sharp of (2) is better monotonically. In order to reduce the calculation amount, only the definition value of the image of the head and shoulder area is calculated, when the sharp is smaller than the definition threshold T sharp Filtering the pedestrian tracking result P i 。
Integrity: p (P) i Intersection area is formed by the rectangular frame of the pedestrian tracking result and the rectangular frames of the rest of the image frame nA inner Calculated to have a maximum value of max (A inner ) Let P be i If the rectangular frame area of (a) is a, the integrity is defined as 1-max (ain)/a, and if the integrity is smaller than the integrity threshold Teclipse, the pedestrian tracking result Pi is filtered.
Brightness: calculation of P i Average gray value gray of (1) when gray is greater than maximum brightness threshold T maxgray Or less than the minimum brightness threshold T mingray Filtering the pedestrian tracking result P i 。
Aspect ratio: assuming that the aspect ratio of a rectangular pedestrian frame accords with normal Gaussian distribution, obtaining the mean value u and standard deviation std of a Gaussian model through big data statistics, and calculating P i The aspect ratio score is higher the smaller the absolute value of the difference of aspect from the mean u, and vice versa. When the aspect ratio score is less than the minimum aspect ratio score threshold T aspect Filtering the pedestrian tracking result P i 。
When the quality evaluation indexes meet the requirements, P i And the next round of filtering process is carried out.
1.3.3 gesture filtration
For P entering the round of filtering i And (3) performing pedestrian gesture evaluation, defining the pedestrian gesture evaluation as a pedestrian attribute classification task, training a 3-attribute classification model with a main network being a lightweight network mobilent, respectively taking the 3 attributes as front gesture, back gesture and side gesture, and finally respectively outputting probability values of the three gestures. In the present round of filtration, P is not directly filtered i Instead, S in the probability value and quality filtering of the front and back poses 1 Weighted summation is carried out to obtain a final pedestrian quality score S final . Thus, the pedestrian picture with the front or back gesture will obtain a higher score, while the pedestrian picture with the side gesture will have a lower score.
If P i Through the three filtering processes, a filtering result is input. If n pedestrian detection results are stored in the result corresponding to id, if n<m, then directly enter P i If n>Comparing the quality scores if P i S of (2) final And if the score is larger than the minimum value of the scores of the n pedestrian detection results, replacing.
1.4 extraction of pedestrian characteristics and pedestrian Properties
The application provides a multi-task model for extracting pedestrian characteristics and pedestrian attributes. Filtering 1.3 sections to obtain a pedestrian target detection result, firstly intercepting and obtaining a pedestrian image through an envelope frame, and then preprocessing to obtain standard data, wherein the standard data comprises size scaling operation and numerical normalization operation of the image; and then importing the standard data into a model, and obtaining output characteristics, namely required pedestrian characteristics after convolution operation of various operators such as a convolution layer, a normalization layer, an activation layer and the like, wherein the characteristics can output pedestrian attribute classification results after a linear classifier.
More specifically, the multitasking model adopts a architecture as a base line, adopts a Resnet34 as a backbone network, introduces a non-local module, replaces an IBN operator with a BN operator, and in the head part of the output head, besides outputting 512-dimensional pedestrian characteristics, a linear classifier branch is additionally arranged to output 15 pedestrian attributes, and the model output formula is as follows:
；
wherein,for inputting an image +.>For multitasking model, ++>For joint output, ++>As a feature of the pedestrian,is a pedestrian attribute. The multitasking model is subjected to light weight treatment similar to the 1.1 section, and the specific steps are as follows: (1) Feeding inThe weight sparse training of the batch norm layer is performed, and then the weight of the batch norm layer is cut out +.>Smaller channels, overall channel cut-out ratio is 30%; (2) performing finetune fine tuning training on the model after clipping; and (3) carrying out int8 quantization when the model is transplanted to the edge equipment. Therefore, a multitasking lightweight model is realized, a model is multipurpose, and the local characteristics of the attribute can be obtained while the global abstract characteristics are obtained. Thus, the high feature extraction capability on the edge equipment can be maintained, and meanwhile, the high calculation efficiency can be maintained.
And extracting pedestrian characteristics and pedestrian attributes from the pedestrian pictures in the filtering result by using the multitasking model. 15 pedestrian attributes: whether to wear hat, whether to wear glasses, whether to wear short sleeves, whether to wear long shirts, whether to wear trousers, whether to wear skirt, whether to carry bags, whether to carry backpacks, whether to be old people, whether to be adults, whether to be children, and whether to be females. These features are not ambiguous and can reflect the local features of the pedestrian.
1.5 multiframe pedestrian feature and attribute fusion
And fusing the extracted multiple pedestrian characteristics and pedestrian attributes for the multiple frames of pedestrian detection results with the same id in the filtering result, so that the data transmission quantity can be reduced, the characteristics of the cross-space-time characteristics can be obtained, and the subsequent recognition accuracy is improved. And obtaining a pedestrian characteristic fusion result in a weighted summation mode. For the pedestrian attribute, as the probability value of each attribute is output by the pedestrian attribute branch, in order to ensure the robustness of the result, attribute judgment is performed by adopting a high-low double-threshold mode. For example, if the output probability P of wearing a cap is high, the threshold is P high A low threshold of P low When p is>=P high If so, determining that the cap is worn (True state); when P high >p>P low When it is determined to be unknown (unknown state); when p is<=P low If so, it is determined that the cap is not worn (False state). For attribute determination of pedestrian detection results of multiple frames, carrying out result statistics on each attribute, and carrying out three statesWhich is higher is the pedestrian attribute fusion result. By fusion, only one pedestrian characteristic result and one pedestrian attribute result are corresponding to one id, and then the information of the id is imported into the database.
1.6 in-library data clustering
The current SORT tracking algorithm can have the same pedestrian id switching condition under shielding or high-speed movement, namely, one pedestrian has a plurality of ids. It is the same person that is actually characterized by multiple id information that is embodied in the database. In order to reduce redundancy of the database, mean shift clustering operation is performed in the database, and data are further fused.
2. Pedestrian re-recognition stage (as in figure 2)
For the person to be inquired, the pedestrian characteristic F of the inquired person is obtained through the flow of section 1.1-1.5 in the same way query And pedestrian attribute A query Then, the pedestrian re-recognition is performed through the following two processes:
2.1 feature alignment
F query And (3) carrying out cosine similarity calculation on pedestrian features corresponding to the ids in the database, and sequencing the results to obtain K similar pedestrians with higher similarity, so that the subsequent processing data volume can be effectively reduced.
2.2 Attribute alignment
For the first K similar pedestrians, acquiring the pedestrian attribute results, and A query And comparing pedestrian attributes. For the kth pedestrian P k 15 attribute results thereof are respectively equal to A query The corresponding attributes are compared. For certain attribute information, if one party is unknown, the comparison of the attribute is ignored; if neither is unknown, when the attribute states are consistent, the comparison of the attributes is ignored as well, and if not, P is filtered k . And finally, the retained similar pedestrians are the final pedestrian re-identification result.
Compared with the prior art, the application has the advantages that: the provided multiple actions ensure the accuracy of the final query effect of the scheme while reducing the calculated amount, and are more suitable for the deployment of edge equipment. The measures adopted are as follows: (1) Evaluating and filtering the pedestrian picture by adopting multiple filtering and a plurality of quality evaluation indexes; (2) application of lightweight algorithm: the method comprises the steps of applying a light target detection model and a multi-task feature extraction model, wherein a tracking algorithm is SORT tracking, (3) multiple information fusion of features and attributes between id and id, so that data transmission quantity and calculation quantity are reduced, (4) the query process is secondary judgment, similar pedestrians are respectively searched and filtered from global features and local attributes, and accuracy of a query result is improved.
The above examples are preferred embodiments of the present application, but the embodiments of the present application are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present application should be made in the equivalent manner, and the embodiments are included in the protection scope of the present application.
Claims (12)
1. The pedestrian re-identification method suitable for the edge equipment deployment is characterized by comprising the following steps of:
s1, obtaining pedestrian envelope frames on each frame of image through a target detection model, endowing each pedestrian envelope frame with an id by utilizing a sort multi-target tracking model, and tracking all the pedestrian envelope frames; filtering the pedestrian envelope frames with the same id;
s2, simultaneously extracting pedestrian features and pedestrian attributes from the pedestrian images through a fastreid model, fusing multi-frame pedestrian features and pedestrian attributes of the same id, importing the fused pedestrian features into a database, adopting mean shift clustering in the database, further fusing the pedestrian features among different ids, removing redundant ids, and finally obtaining pedestrian features and attribute information of different ids;
s3, performing cosine distance calculation on pedestrian features of different ids to obtain K similar pedestrians, and then performing secondary filtering by using pedestrian attribute information of the K similar pedestrians to filter attribute mismatch results in the K similar pedestrians so as to obtain the most similar pedestrian re-recognition result.
2. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 1, wherein in step S1, the target detection model is a lightweight yolox_s model.
3. The pedestrian re-identification method suitable for edge device deployment according to claim 2, wherein the lightweight processing of the yolox_s model comprises:
(1) Performing batch norm weight sparse training on the yolox_s model, and then cutting a channel with the bn layer weight of the yolox_s model smaller than a preset value;
(2) Performing finetune fine tuning training on the trimmed yolox_s model;
(3) The yolox_s model performs int8 quantization when the edge device is migrated.
4. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 3, wherein the bn layer of the yolox_s model is a feature normalization operator of the yolox_s model, and the feature normalization operator is a batch norm2d operator, and the formula is as follows:
；
wherein,for input data requiring normalization, +.>And->For the mean and variance of the batch data, +.>Variable added to prevent zero occurrence of denominator, < >>And->Affine operation, namely linear transformation is carried out on the input value; from the formula, the weight of bn layer is +.>And->Output->Each channel in (a) has a corresponding +.>And->Use +.>Judging the importance degree of different channels; when->And if the weight of the channel is smaller than the preset value, the weight of the channel is cut if the data of the channel has smaller influence on the follow-up model.
5. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 1, wherein in step S1, the working process of the target detection model is as follows:
firstly, acquiring image data from a camera terminal, preprocessing the image data, and inputting the preprocessed image data into a target detection model yolox_s for reasoning: the image data is subjected to convolution operation on a backbone network of a target detection model yolox_s to obtain an output characteristic diagram; then inputting the feature map to an output head of a target detection model yolox_s to carry out regression and classification of a target frame;
the output head of the object detection model yolox_s has three branches in total: cls_output is used for predicting the class and the fraction of the target frame, the output size is W.H.C, wherein W is the width of the output feature map, H is the height of the output feature map, and C is the number of detection classes; the obj_output is used for judging whether the target frame is a foreground or a background, and the output size is W.times.H.times.1; reg_output used for predicting coordinate information (x, y, W, H) of the target frame, and outputting a W x 4;
combining three branches of an output head of a target detection model yolox_s to obtain final candidate prediction frame information, wherein the size of the final candidate prediction frame information is pred_num_dim, wherein pred_num=w_h, the number of the prediction frames is represented, dim=c+1+4, and the feature vector dimension of each prediction frame is represented; each prediction frame contains a feature vector with a dimension dim, [ cls_1 cls_2 ]. Cls_C obj x y h ], wherein x is x coordinate information of a center point of the target frame, y is y coordinate information of the center point of the target frame, w is width information of the target frame, h is height information of the target frame, obj is score information of the target frame, cls_1 is score of a category 1 of the target frame, and cls_C is score of a category C of the target frame;
post-processing candidate prediction frame results: firstly, reducing the normalized coordinates of the predicted frame to the actual pixel size, then filtering the redundant predicted frame by utilizing non-maximum value inhibition, and outputting to obtain the final pedestrian envelope frame.
6. The pedestrian re-identification method suitable for edge device deployment of claim 5, wherein the pre-processing includes normalized scaling of image size and normalization of data.
7. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 1, wherein in step S1, the working process of the sort multi-target tracking model is as follows:
(1) An id is assigned to each pedestrian envelope frame obtained by the target detection model;
(2) Initializing a target tracker: for each detected pedestrian target, creating a Kalman filtering target tracker for each detected pedestrian target, wherein the Kalman filtering target tracker is used for tracking the motion of the pedestrian target in the subsequent image frames;
(3) Multi-frame tracking: for each frame, the following operations are performed:
1) And (3) data association: matching a pedestrian envelope frame bboxmeasurement obtained by a Kalman filtering prediction pedestrian envelope frame bboxprediction and a target detection model by using a Hungary algorithm;
2) And (5) updating the state: and updating the current state of a pedestrian envelope frame bboxmeasurement obtained by using a Kalman filtering prediction pedestrian envelope frame bboxprediction and a target detection model to obtain a pedestrian tracking envelope frame bboxoptimal as a tracking result.
8. The pedestrian re-recognition method suitable for edge device deployment according to claim 1, wherein in step S1, the filtering includes space-time filtering, quality filtering, and pose evaluation performed sequentially; the space-time filtering means that for a certain pedestrian tracking result with existing id, only the pedestrian tracking result with the frame interval larger than the time threshold and the space interval larger than the space threshold of the previous frame image is reserved, and then the reserved pedestrian tracking result enters the quality filtering; the quality filtering means filtering the pedestrian tracking results which do not meet the quality index, and scoring each quality index of the pedestrian tracking results at the same time, wherein the weighted summation of all the quality index scores is the quality score S of the pedestrian tracking results 1 Then, the reserved pedestrian tracking result enters into gesture evaluation; the gesture evaluation is to divide the pedestrian gesture into a plurality of attributes, and output probability values of the plurality of pedestrian gestures and quality scores S through an attribute classification model 1 Weighted summation to obtain final pedestrian quality score S final ；
Inputting a pedestrian tracking result through space-time filtering, quality filtering and attitude evaluation: if the corresponding id in the pedestrian tracking result is stored with n pedestrian tracking results, if n is smaller than the frame number corresponding to the id, directly inputting, if n is greater than or equal to the frame number corresponding to the id, comparing the pedestrian quality scores S final Pedestrian quality score S if pedestrian tracking result final And if the score is larger than the minimum value of the scores of the n pedestrian tracking results, replacing.
9. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 1, wherein in step S2, pedestrian features and pedestrian attributes are simultaneously extracted from the pedestrian image through a fastreid model, specifically as follows: firstly, intercepting pedestrians in an image through a reserved pedestrian envelope frame, and then, preprocessing to obtain standard data; and then importing the standard data into a fastreid model, obtaining pedestrian characteristics after convolution operation, and outputting pedestrian attribute classification results after the pedestrian characteristics pass through a linear classifier.
10. The pedestrian re-recognition method suitable for edge equipment deployment according to claim 1, wherein in step S2, the multi-frame pedestrian feature fusion is obtained by a weighted summation mode; the multi-frame pedestrian attribute adopts a high-low double-threshold mode to judge the attribute, and performs result statistics on each attribute, and the state with the highest accumulated value is the pedestrian attribute fusion result; by fusing, one id corresponds to only one pedestrian feature result and one pedestrian attribute result.
11. A server comprising a processor and a memory, wherein the memory has stored therein at least one program that is loaded and executed by the processor to implement the pedestrian re-identification method of any one of claims 1 to 10 that is suitable for edge device deployment.
12. A computer readable storage medium, characterized in that at least one program is stored in the storage medium, which program is loaded and executed by a processor to implement the pedestrian re-identification method suitable for edge device deployment according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311450373.4A CN117173794B (en) | 2023-11-03 | 2023-11-03 | Pedestrian re-identification method suitable for edge equipment deployment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311450373.4A CN117173794B (en) | 2023-11-03 | 2023-11-03 | Pedestrian re-identification method suitable for edge equipment deployment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117173794A true CN117173794A (en) | 2023-12-05 |
| CN117173794B CN117173794B (en) | 2024-03-08 |
Family
ID=88939869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311450373.4A Active CN117173794B (en) | 2023-11-03 | 2023-11-03 | Pedestrian re-identification method suitable for edge equipment deployment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117173794B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115837A (en) * | 2020-09-11 | 2020-12-22 | 中国电子科技集团公司第五十四研究所 | Target detection method based on YoloV3 and dual-threshold model compression |
| CN112818917A (en) * | 2021-02-24 | 2021-05-18 | 复旦大学 | Real-time pedestrian detection and re-identification method and device |
| CN114863366A (en) * | 2022-05-30 | 2022-08-05 | 上海数川数据科技有限公司 | Cross-camera multi-target tracking method based on prior information |
| CN115049607A (en) * | 2022-06-10 | 2022-09-13 | 四川轻化工大学 | Insulating plate defect identification method based on YOLOx _ s enhanced target feature detection |
| US20220383662A1 (en) * | 2020-01-10 | 2022-12-01 | Sportlogiq Inc. | System and Method for Identity Preservative Representation of Persons and Objects Using Spatial and Appearance Attributes |
| CN116012880A (en) * | 2022-12-29 | 2023-04-25 | 天翼物联科技有限公司 | Pedestrian re-identification method, system and device for distributed edge collaborative reasoning |
-
2023
- 2023-11-03 CN CN202311450373.4A patent/CN117173794B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220383662A1 (en) * | 2020-01-10 | 2022-12-01 | Sportlogiq Inc. | System and Method for Identity Preservative Representation of Persons and Objects Using Spatial and Appearance Attributes |
| CN112115837A (en) * | 2020-09-11 | 2020-12-22 | 中国电子科技集团公司第五十四研究所 | Target detection method based on YoloV3 and dual-threshold model compression |
| CN112818917A (en) * | 2021-02-24 | 2021-05-18 | 复旦大学 | Real-time pedestrian detection and re-identification method and device |
| CN114863366A (en) * | 2022-05-30 | 2022-08-05 | 上海数川数据科技有限公司 | Cross-camera multi-target tracking method based on prior information |
| CN115049607A (en) * | 2022-06-10 | 2022-09-13 | 四川轻化工大学 | Insulating plate defect identification method based on YOLOx _ s enhanced target feature detection |
| CN116012880A (en) * | 2022-12-29 | 2023-04-25 | 天翼物联科技有限公司 | Pedestrian re-identification method, system and device for distributed edge collaborative reasoning |
Non-Patent Citations (2)
| Title |
|---|
| ZHANG LILIANG ET AL: "Is faster RCNN doing well for pedestrian detection", 《EUROPEAN CONFERENCE ON4 COMPUTER VISION》, 31 December 2016 (2016-12-31), pages 443 - 457 * |
| 韩俊 等: "基于特征增强的多尺度SSD行人检测算法", 《信息技术与信息化》, no. 1, 10 February 2020 (2020-02-10), pages 117 - 121 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117173794B (en) | 2024-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109934176B (en) | Pedestrian recognition system, recognition method, and computer-readable storage medium | |
| Kawewong et al. | Online and incremental appearance-based SLAM in highly dynamic environments | |
| CN109635686B (en) | Two-stage pedestrian searching method combining human face and appearance | |
| CN110717411A (en) | Pedestrian re-identification method based on deep layer feature fusion | |
| CN110263712B (en) | Coarse and fine pedestrian detection method based on region candidates | |
| CN110569754A (en) | Image target detection method, device, storage medium and equipment | |
| WO2022198817A1 (en) | Vehicle image clustering method and vehicle trajectory restoration method | |
| Zhan et al. | Pedestrian detection and behavior recognition based on vision | |
| CN109359530B (en) | Intelligent video monitoring method and device | |
| CN116824641B (en) | Gesture classification method, device, equipment and computer storage medium | |
| CN113723188A (en) | Dress uniform person identity verification method combining face and gait features | |
| CN111160099B (en) | Intelligent segmentation method for video image target | |
| CN112509009A (en) | Target tracking method based on natural language information assistance | |
| CN114463800A (en) | Multi-scale feature fusion face detection and segmentation method based on generalized intersection-parallel ratio | |
| CN117173794B (en) | Pedestrian re-identification method suitable for edge equipment deployment | |
| CN116311518A (en) | Hierarchical character interaction detection method based on human interaction intention information | |
| Park et al. | Intensity classification background model based on the tracing scheme for deep learning based CCTV pedestrian detection | |
| CN115512263A (en) | Dynamic visual monitoring method and device for falling object | |
| CN115565157A (en) | Multi-camera multi-target vehicle tracking method and system | |
| WO2021259033A1 (en) | Facial recognition method, electronic device, and storage medium | |
| CN115457595A (en) | Method for associating human face with human body, electronic device and storage medium | |
| CN113627383A (en) | Pedestrian loitering re-identification method for panoramic intelligent security | |
| Khosla et al. | A neuromorphic system for object detection and classification | |
| CN113361475A (en) | Multi-spectral pedestrian detection method based on multi-stage feature fusion information multiplexing | |
| CN117576764B (en) | Video irrelevant person automatic identification method based on multi-target tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |