CN114898402A - Pedestrian re-identification method, device and medium fusing human face and human body characteristics - Google Patents
Pedestrian re-identification method, device and medium fusing human face and human body characteristics Download PDFInfo
- Publication number
- CN114898402A CN114898402A CN202210460570.3A CN202210460570A CN114898402A CN 114898402 A CN114898402 A CN 114898402A CN 202210460570 A CN202210460570 A CN 202210460570A CN 114898402 A CN114898402 A CN 114898402A
- Authority
- CN
- China
- Prior art keywords
- human body
- feature
- face
- information
- similarity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Collating Specific Patterns (AREA)
Abstract
The invention provides a pedestrian re-identification method fusing human face and multi-angle human body characteristics, which comprises the following steps: s1, acquiring a face snapshot image and a human body snapshot image, performing quality evaluation on the face snapshot image and the human body snapshot image, then extracting face characteristics and human body characteristics, and acquiring first characteristic information and second characteristic information; the first feature information comprises face feature information and face attribute information, and the second feature information comprises human body feature information and human body attribute information; s2, comparing the first characteristic information and the second characteristic information which are used as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix; and S3, matching the identity of the candid person based on the similarity matrix to output the matched person identity ID. The pedestrian re-identification method can improve the identification accuracy.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a pedestrian re-identification method, a device and a medium fusing human face and human body characteristics.
Background
The pedestrian re-identification method based on deep learning mainly comprises two methods: the first method is to construct a deep neural network for extracting human body features, obtain a feature extractor with higher precision through training of a large amount of labeled data, directly input a human body image to be recognized into the network to obtain features in a recognition stage, and then compare the features, wherein the features with high similarity are the same person; the second type of re-recognition is based on pedestrian attribute matching, and mainly comprises the steps of splitting human body features, such as attributes of hair, jacket, lower body, shoes, whether a backpack is needed, whether a hat is needed or not, and the like, constructing a deep neural network to directly recognize the attribute features of the human body, and then completing the re-recognition task of the pedestrian by combining the similarity of the attributes. The two pedestrian re-identification schemes are innovative around the aspects of improvement and optimization of the deep neural network. In the face of a pedestrian re-identification task in a complex scene, the complexity of a neural network model is continuously increased, the identification precision can be improved to a certain extent, for example, on a Market1501 data set in an outdoor open scene, a deep neural network model already reaches 90% precision, but the capability of generalizing the algorithm to other scenes is very limited due to the fact that the application of a pedestrian re-identification algorithm has special requirements, such as the problems of open scenes, non-matching type, low resolution, different shooting angles and the like, and even if the precision of 90% is reached on the Mark1501, the requirement of application can be met in an actual process.
Reference 1(CN112347957A) discloses a pedestrian re-identification method, which obtains images of multiple pedestrians, performs feature extraction on the images by using a feature extraction model, stores the images in a pedestrian feature library, updates features of the pedestrian feature library according to a preset frequency, performs feature extraction on the images containing the pedestrians to be identified to obtain features to be identified in an identification stage, compares the features with features stored in the pedestrian feature library, selects a most similar target feature from the result, and determines the identity of the pedestrian to be identified according to the ID of the target feature.
Most of the existing pedestrian re-identification models based on the deep neural network directly extract pedestrian features, and then the pedestrian features are compared with the ID of the person in the bank, so that the identity of the person to be identified can be determined. However, the effect is not particularly good in view of the effect of the current practical engineering application. Firstly, training a high-precision feature extraction model, needing a large amount of labeled pedestrian data, and the pedestrian data provided by the current public data set cannot reach an ideal state; secondly, aiming at an end-to-end model of a pedestrian re-identification task, high precision necessarily means high complexity, and the model is necessarily overlarge, so that performance indexes such as real-time performance, memory occupation and the like are often not achieved in actual engineering application; finally, even if a model meeting the performance and index requirements is obtained, the generalization capability of the model is difficult to withstand the practical tests of engineering, and under the comprehensive action of factors such as illumination difference of open scenes, uneven imaging quality of different cameras, multi-end change of pedestrian postures and the like, the single model is difficult to meet the practical requirements of engineering.
The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, material described in this section is not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.
Disclosure of Invention
Aiming at the technical problems in the related art, the invention provides a pedestrian re-identification method fusing human face and multi-angle human body characteristics, which comprises the following steps:
s1, acquiring a face snapshot image and a human body snapshot image, and extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image to acquire first characteristic information and second characteristic information; the first feature information comprises face feature information and face attribute information, and the second feature information comprises human body feature information and human body attribute information;
s2, comparing the first characteristic information and the second characteristic information serving as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix;
and S3, matching the identity of the candid person based on the similarity matrix to output the matched person identity ID.
Specifically, the generating the similarity matrix specifically includes:
s21, obtaining the zeroth similarity of the TOP1 result by the obtained first characteristic information or second characteristic information and a preset first characteristic library;
s22, acquiring the identity ID of the TOP1, and acquiring the identity ID feature corresponding to the identity ID from the rest feature library;
and S23, performing similarity calculation based on the first characteristic information and the second characteristic information and the ID characteristic, and respectively obtaining a first similarity, a second similarity, a third similarity and a fourth similarity.
Specifically, the feature degrees are clustered to obtain the feature center.
Specifically, the method further includes, before the step S2, step S11 of performing eligibility check on the face snapshot image and the human body snapshot image; and extracting the face features and the human body features of the qualified face snapshot image and human body snapshot image to obtain first feature information and second feature information.
Specifically, the step S3 includes: and determining a corresponding characteristic library according to the human body angle information in the second characteristic information, and determining the personnel identity ID according to the determined corresponding characteristic library.
Specifically, the human body attribute information includes: the human body angles comprise a front side, a back side and side surfaces, and the human body postures comprise a standing posture, a sitting posture and a lying posture; the face attributes include whether the face is wearing a mask.
In a second aspect, another embodiment of the present invention discloses a pedestrian re-identification apparatus fusing human face and multi-angle human body features, which includes the following units:
the characteristic information acquisition unit is used for acquiring a face snapshot image and a human body snapshot image, extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image and acquiring first characteristic information and second characteristic information; the first characteristic information comprises face characteristic information and face attribute information, and the second characteristic information comprises human body characteristic information and human body attribute information;
the similarity matrix generating unit is used for comparing the first characteristic information and the second characteristic information serving as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix;
and the pedestrian re-identification unit is used for matching the identities of the snap-shot personnel based on the similarity matrix so as to output the matched personnel identity IDs.
Specifically, the similarity matrix generating unit further includes:
the first feature matching unit is used for obtaining the zeroth similarity of the TOP1 result by using the acquired first feature information or second feature information and a preset first feature library;
an identity ID obtaining unit, configured to obtain an identity ID of the TOP1, and obtain identity ID features corresponding to the identity ID from the remaining feature library;
and the second feature matching unit is used for performing similarity calculation on the basis of the first feature information and the second feature information and the ID feature to respectively obtain a first similarity, a second similarity, a third similarity and a fourth similarity.
Specifically, the feature degrees are clustered to obtain the feature center.
Specifically, the pedestrian re-identification unit includes: and determining a corresponding characteristic library according to the human body angle information in the second characteristic information, and determining the personnel identity ID according to the determined corresponding characteristic library.
In a third aspect, another embodiment of the present invention discloses a non-volatile memory, where the memory stores instructions, and the instructions are executed by a processor, and are used to implement the above pedestrian re-identification method that integrates human face and multi-angle human body features.
According to the invention, the human face and the multi-angle human body features are fused, so that firstly, the accuracy of identity recognition by simply using the human body features is effectively improved; further, 5 feature libraries are constructed, real-time updating is carried out on the feature libraries, and periodic clustering is carried out to obtain feature centers of each ID in each library; secondly, the real-time updating of 5 feature libraries enables the features closest to the current space-time to be put in storage in time, the feature center obtained by periodic clustering overcomes the problem of the change of the same ID feature under the condition of crossing space-time, and the comprehensive application of the feature center and the feature library ensures the high reliability of identity identification; the invention discloses an identity identification method based on a similarity matrix, which can obviously improve the identification accuracy. The pedestrian re-identification method disclosed by the invention integrates and applies multi-angle human body characteristics, and is very suitable for the actual situation of various monitoring scenes at present, because a high-quality front human body or human face image is not obtained under any condition in the air, and because the human face grab-shooting and the human body grab-shooting with the same ID are bound, when the human body posture does not have an identification condition or the human body characteristics with multiple IDs have higher similarity, the identity recognition can be still carried out by utilizing the human face grab-shooting, so that the problem of identity misrecognition and rejection under the conditions of various human body postures or different people with the same robe, different robes with the same person and the like is effectively solved, and the actual combat effect of pedestrian re-recognition is greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a pedestrian re-identification method integrating human face and multi-angle human body features according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a similarity matrix provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a similarity matrix calculation process provided by the embodiment of the present invention;
FIG. 4 is a schematic diagram of a feature center generation process provided by an embodiment of the present invention;
FIG. 5 is a schematic diagram of a pedestrian re-identification system fusing human face and multi-angle human body features according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a pedestrian re-identification apparatus fusing human face and multi-angle human body features according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a pedestrian re-identification device fusing human face and multi-angle human body features according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
Example one
Referring to fig. 1 and 5, the embodiment discloses a pedestrian re-identification method fusing human faces and multi-angle human body features, which includes the following steps:
s1, acquiring a face snapshot image and a human body snapshot image, and extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image to acquire first characteristic information and second characteristic information; the first feature information comprises face feature information and face attribute information, and the second feature information comprises human body feature information and human body attribute information;
in the step, a face snapshot image can be obtained through a special face snapshot camera, wherein the face snapshot image is an image concentrated on a face part, and facial features of a person, such as feature information of the face, information about whether a mask is worn and the like, can be obtained from the face snapshot image.
The specific first feature information not only includes face feature information, but also includes face attribute information. The face attribute information includes, but is not limited to, whether a mask is worn, hair color, and the like.
Common facial features are divided into two categories, one is geometric features and the other is characterization features. Geometric features refer to the geometric relationships between facial features such as eyes, nose, and mouth, such as distance, area, and angle. The characterization feature utilizes gray information of the face image.
In the step, a human body snapshot camera can be used for acquiring a human body snapshot image, wherein the human body snapshot image is an image containing a main human body, and the facial features, postures and other feature information of a person can be acquired from the human body snapshot image.
The second feature information includes not only the human body feature information but also human body attribute information.
The commonly used human characteristic information is mainly human key point information, for example, the COCO data set represents the human key points as 17 joints, which are respectively a nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.
The body attribute information includes, but is not limited to, body angle, body posture, clothing color, and the like.
Specifically, the present embodiment uses a deep learning manner to extract the face feature information and the body feature information.
In this embodiment, a face feature and body feature information extraction network is trained in advance to extract face feature information and body feature information. The specific training method of this embodiment is not further limited, and a training mode of convolutional neural network known in the art may be used.
Further, the implementation may also train a face feature extraction network and a body feature extraction network respectively for extracting face feature information and body feature information.
Specifically, the present embodiment integrates the above-mentioned facial feature and human body feature information extraction networks into a basic information extraction module. The basic information extraction module is used for extracting the face characteristic information and the human body characteristic information.
The human face snapshot image and the human body snapshot image are input into a basic information extraction module, the module comprises human face feature extraction, human face mask wearing judgment, human face quality evaluation, human body feature extraction and human body angle judgment, the human body angle judgment is used for identifying whether a human body is a front face, a side face or a back face and judging human body postures, and the human body posture judgment is used for identifying whether the human body stands, sits or lies.
After the basic information extraction module, the face snapshot image and the human body snapshot image not only extract the features of the face and the human body, but also add other attribute information, such as whether the face wears a mask, the angle of the human body, the posture of the human body and the like, and the attributes play a role in auxiliary judgment on identity recognition.
This embodiment carries out identification through fusing people's face and multi-angle human characteristic information, can further improve the accuracy of heavy discernment.
Each sub-module in the basic information extraction module can be generally constructed by adopting a deep neural network model, and the deep neural network model is not limited to this, as long as the functions can be realized, and the face feature extraction and the human body feature extraction can be realized.
Furthermore, when the human body in the shooting range is the back or the side with a large angle, that is, the human face cannot be shot, the embodiment does not limit that the human body snapshot image must contain the human face part; when the front human body is in the shooting range, but the human body is far away from the camera, the imaging is small, the human face may be missed, at the moment, the human face detection is performed again in the process of capturing the picture from the front human body, and the detected human face and the human body are bound.
S2, comparing the first characteristic information and the second characteristic information serving as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix;
specifically, there are 5 feature libraries preset in this embodiment. The 5 feature libraries are a face registration photo feature library, a face grabbing photo feature library, a front human body feature library, a side human body feature library and a back human body feature library respectively;
specifically, the data of the feature library may be pre-stored corresponding data, for example, data already existing in import.
Or the data of the feature library, may be gradually added in the process of executing the method of the present embodiment. For example, as pedestrians enter a monitored area and start moving, cameras deployed at various point locations continuously capture faces and human bodies, and data is gradually added to the feature library, so that as the pedestrians enter the monitored area for a longer time, the data accumulated in the feature library is richer and richer, and the accuracy of pedestrian re-identification is certainly higher and higher.
The difference between the face registration photo and the face capture photo in the implementation is that the former is to extract and store the features of a high-quality face image selected from the capture photo when a pedestrian just enters a monitoring area, for example, if a service scene of personnel cooperation exists in some areas, for example, a case execution area, a higher-quality face image can be shot from a matching link, and then the features are extracted and enter a face registration photo feature library.
Specifically, the present embodiment may perform eligibility check on the face snapshot image and the human body snapshot image, so as to improve accuracy of re-recognition, and when the output face features are not qualified, for example, the face snapshot image is not clear, although face feature information may be output, the output feature information is not correct, which may affect accuracy of subsequent re-recognition.
The human face quality evaluation in the basic information extraction module is used for checking the qualification of the snapshot human face, and the human body quality evaluation module is used for checking the qualification of the snapshot human body.
The embodiment mainly judges whether the human face and the human body are fuzzy or not and whether the human face and the human body are large-angle or not through the human face key points and the human body key points so as to be unsuitable for identification to remove unqualified images.
The method comprises the following specific steps that before the step S2, S11 is further included before the step S2, and the qualification checks are carried out on the face snapshot image and the human body snapshot image; and extracting the face features and the human body features of the qualified face snapshot image and human body snapshot image to obtain first feature information and second feature information.
In this embodiment, the similarity matrix is used for pedestrian re-identification, specifically, the calculation method of the similarity matrix is as follows:
referring to fig. 2, fig. 2 is a similarity matrix of the present embodiment, the head on the left of the matrix represents the top1 similarity value that the current captured image matches in the bottom library corresponding to its type, the head on the top represents the ID corresponding to the top1, and the similarity values between the feature centers in the other 4 feature libraries and the feature to be identified currently. Wherein, the register _ face is a face registration photo feature library, the capture _ face is a face fear photo feature library, the front _ body is a front human body feature library, the side _ body is a side human body feature library, and the back _ body is a back human body feature library; the register _ face _ top1 is a matching top1 similarity value in a face registered photo feature library, the capture _ face _ top1 is a matching top1 similarity value in a face scratch photo feature library, the front _ body _ top1 is a matching top1 similarity value in a front human feature library, the side _ body _ top1 is a matching top1 similarity value in a side human feature library, and the back _ body _ top1 is a matching top1 similarity value in a back human feature library.
The calculation method of the similarity matrix in this embodiment is as follows:
s21, obtaining the zeroth similarity of the TOP1 result by the obtained first characteristic information or second characteristic information and a preset first characteristic library;
specifically, for a matching top1 similarity value in a feature library, for example, for a matching top1 similarity value in a face registration photo feature library, obtaining first feature information and matching the first feature information with the face registration photo feature library to obtain a top1 similarity value, and then the preset first feature library in this step is a face registration feature library;
for the matched top1 similarity value in the face snapshot feature library, acquiring first feature information, matching the first feature information with the face snapshot feature library to obtain a top1 similarity value, and using the preset first feature library in the step as a face snapshot feature library;
s22, acquiring the identity ID of the TOP1, and acquiring the identity ID feature corresponding to the identity ID from the rest feature library;
and S23, calculating similarity based on the first characteristic information, the second characteristic information and the ID characteristic, and respectively obtaining a first similarity, a second similarity, a third similarity and a fourth similarity.
In the embodiment, when the snapshot is a human face and a front human body, the first row of the similarity matrix is taken as an example, and the top1 similarity value matched in the feature library of the human face registration photo in fig. 2 is referred to.
The specific calculation flow is shown in fig. 3, and mainly includes 3 steps:
the first step is as follows: matching the characteristics of the current face snapshot with a face registry to obtain top1, wherein the similarity 0 is the similarity value between the characteristics of the face to be recognized and the matched top1 ID 1;
the second step is that: respectively acquiring the characteristics corresponding to the ID1 in other 4 characteristic libraries;
the third step: and respectively carrying out similarity calculation on the current face snapshot feature and the front human body snapshot feature and corresponding features to obtain a similarity 1, a similarity 2, a similarity 3 and a similarity 4. By this point, the calculation of the first row in the similarity matrix is finished, and the other rows are similar.
Therefore, the values on the diagonal line from the upper left corner to the lower right corner of the similarity matrix are similarity values of top1 obtained by matching the snapshot features in the corresponding feature library, and the rest values in each row are similarity values of the feature centers of the IDs corresponding to top1 in other libraries and the current to-be-identified corresponding type snapshot features. It should be emphasized that the type of the captured image is particularly emphasized in the above steps because the face captured image only performs similarity calculation with two face feature libraries, and the human body captured image only performs similarity calculation with 3 human body feature libraries.
With the continuous activities of pedestrians in the monitored area, the number of snapshots is continuously increased, the number of features in the feature library is increased, however, in consideration of the problem of resource consumption, the feature library cannot be traversed and compared with all in-library features of all IDs every time, and therefore, the existence of the feature center is particularly important.
The feature center refers to a feature in the current feature library that is closest to the sum of distances of other features. It should be noted that, the clustering method for the feature centers in this embodiment may adopt, but is not limited to, K-Means clustering, as long as the clustering method can obtain the feature centers meeting the above definition.
The clustering process of the feature center is shown in fig. 4, when a face and a human body are detected by each camera, the camera continuously performs snapshot in a fixed period, the snapshot obtains face/human body features after basic information extraction, and once the identity is determined, the features are stored in one of face or human body feature libraries corresponding to the current ID through identity recognition based on a similarity matrix. Then, each feature library is clustered according to a fixed period, so that the feature center of each feature library is continuously refreshed. And each feature library carries out clustering analysis according to a certain period to form a real-time feature center.
The capture pictures in fig. 4 are taken every 3 seconds, and the feature library is clustered every 10 minutes, both of which are empirical values obtained from many experiments in a specific scenario, and if the scenarios are different, the settings of the above parameters may need to be adjusted.
And S3, matching the identity of the snap shot personnel based on the similarity matrix to output the matched personnel identity ID.
Specifically, in this embodiment, a corresponding feature library is determined according to the human body angle information in the second feature information, and the person identity ID is determined according to the determined corresponding feature library.
Specifically, in one embodiment, for a human body with a front face, the corresponding feature library is a register _ face registration photo feature library, a capture _ face grabbing photo feature library, or a front _ body front face human body feature library; and determining the personnel identity ID according to the three feature libraries of the register _ face, the capture _ face and the front _ body which correspond to the determination.
In another embodiment: at the current moment, a human face feature A and a back human face feature B exist, firstly, A and a human face registry are used for matching to obtain the similarity D1 corresponding to top1, meanwhile, the ID1 corresponding to top1 is determined, the ID1 is used for finding 4 features C, D, E, F in the rest 4 feature libraries respectively, the similarity D2 is obtained through A and C, and the similarities D3, D4 and D5 are obtained through B and D, E, F respectively. The calculation flows of the other rows in the similarity matrix are similar, a point is clear, and the similarities on the diagonal lines from the upper left corner to the lower right corner are the similarities corresponding to top1 of A and B in the 5 feature libraries.
Identity matching logic:
1. if D1 is greater than the threshold value of the face similarity, the snapshot can be preliminarily judged to belong to ID1, and if D2 and D5 are also greater than the threshold values respectively, the snapshot belongs to ID 1;
2. if D1 is not greater than the threshold, D2 will not be greater than the threshold, and when D5 is greater than the threshold, the identity of the captured picture can also be determined from D5.
The similarity matrix comprehensively reflects the matching conditions of the face to be recognized, the human body and all in-bank IDs, and can conveniently complete the task of re-recognizing pedestrians based on the matrix, and the similarity matrix mainly comprises the following conditions:
1. the snapshot is of human face and human body in front: the method comprises the steps of taking a matching result of a face registration feature library as a main part, taking the similarity of a front human body feature library as correction, and determining the identity of a person to be recognized when two similarity values are greater than respective threshold values; specifically, if the snapshot is a human face and a human body on the front side, in a general case, the similarity of the human face (that is, the value of the 0 position in the similarity matrix) is greater than the threshold, and if the value of the 2 position in the similarity matrix is also greater than the threshold, the identity of the person can be given.
Each position of the similarity matrix in this embodiment has its own threshold.
2. The snapshot is human face and side human body: at the moment, the human face features are influenced by self-shielding, the matching results of the side human body feature library are taken as the main, and when the similarity is greater than respective threshold values, the identity of a person to be recognized can be determined;
3. the back human body is captured and photographed: when the human face information is lacked, the recognizable features of the human body on the back are less, and as long as the similarity of the human body is greater than a set threshold value, a recognition result is given.
In the above case, the most difficult recognition result is the third one, but the result is equivalent to the effect of the existing back identification method which only depends on the human body characteristics, and in the other two cases, the method of the embodiment is logically better than the scheme which only depends on the human body characteristics.
In addition, the similarity matrix provides a lot of information that can be used in more complex cases:
1. strange person identical robe:
the strange robe refers to two different people, but the clothes are similar, and the strange identification is easy to be realized by the simple dependence on the human body characteristics. However, in the present solution, when a face is captured, the face features of two IDs inevitably have a large difference, so that this situation can be distinguished more easily with reference to the similarity matrix.
2. Identical person's special robe
The identical person and the different gowns refer to the same person, but are different in clothes under different space-time scenes, and the situation is also difficult and pain points when the pedestrian is identified again only by human body characteristics at present. Similarly, when a face is captured, although the difference of human body characteristics is large due to different dresses, according to the matching logic in the scheme, the face characteristics are taken as the main part, and an accurate result can still be given by referring to the similarity matrix.
According to the embodiment, the human face and the multi-angle human body features are fused, so that firstly, the accuracy of identity recognition by simply using the human body features is effectively improved; further, 5 feature libraries are constructed, real-time updating is carried out on the feature libraries, and periodic clustering is carried out to obtain feature centers of each ID in each library; secondly, the real-time updating of 5 feature libraries enables the features closest to the current space-time to be put in storage in time, the feature center obtained by periodic clustering overcomes the problem of the change of the same ID feature under the condition of crossing space-time, and the comprehensive application of the feature center and the feature library ensures the high reliability of identity identification; the embodiment discloses an identity recognition method based on a similarity matrix, which can obviously improve the recognition accuracy. The pedestrian re-identification method of the embodiment is integrated with and applies multi-angle human body features, and is very suitable for actual conditions of various current monitoring scenes, because high-quality front human body or human face images are not acquired under any conditions in the air, and because the human face capture and the human body capture of the same ID are bound, when human body postures do not have identification conditions or human body features of multiple IDs have higher similarity, the identity identification can be still performed by utilizing the human face capture, so that the problem of identity misrecognition and rejection under the conditions of various human body postures or different people with one robe, different people with one robe and the like is effectively solved, and the actual combat effect of pedestrian re-identification is greatly improved.
Example two
Referring to fig. 6, the present embodiment discloses a pedestrian re-identification apparatus fusing human face and multi-angle human body features, which includes the following units:
the characteristic information acquisition unit is used for acquiring a face snapshot image and a human body snapshot image, extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image and acquiring first characteristic information and second characteristic information; the first characteristic information comprises face characteristic information and face attribute information, and the second characteristic information comprises human body characteristic information and human body attribute information;
the similarity matrix generating unit is used for comparing the first characteristic information and the second characteristic information serving as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix;
there are 5 feature libraries preset in this embodiment. The 5 feature libraries are a face registration photo feature library, a face grabbing photo feature library, a front human body feature library, a side human body feature library and a back human body feature library respectively;
in this embodiment, the similarity matrix is used for pedestrian re-identification, specifically, the calculation method of the similarity matrix is as follows:
referring to fig. 2, fig. 2 is a similarity matrix of the present embodiment, the head on the left of the matrix represents the top1 similarity value that the current captured image matches in the bottom library corresponding to its type, the head on the top represents the ID corresponding to the top1, and the similarity values between the feature centers in the other 4 feature libraries and the feature to be identified currently. Wherein, the register _ face is a face registration photo feature library, the capture _ face is a face-grabbing photo feature library, the front _ body is a front human body feature library, the side _ body is a side human body feature library, and the back _ body is a back human body feature library; the register _ face _ top1 is a matching top1 similarity value in a face registered photo feature library, the capture _ face _ top1 is a matching top1 similarity value in a face scratch photo feature library, the front _ body _ top1 is a matching top1 similarity value in a front human feature library, the side _ body _ top1 is a matching top1 similarity value in a side human feature library, and the back _ body _ top1 is a matching top1 similarity value in a back human feature library.
The calculation method of the similarity matrix in this embodiment is as follows:
the first feature matching unit is used for obtaining the zeroth similarity of the TOP1 result by using the acquired first feature information or second feature information and a preset first feature library;
specifically, for a matching top1 similarity value in a feature library, for example, for a matching top1 similarity value in a face registration photo feature library, obtaining first feature information and matching the first feature information with the face registration photo feature library to obtain a top1 similarity value, and then the preset first feature library in this step is a face registration feature library;
for the matched top1 similarity value in the face snapshot feature library, acquiring first feature information, matching the first feature information with the face snapshot feature library to obtain a top1 similarity value, and using the preset first feature library in the step as a face snapshot feature library;
an identity ID obtaining unit, configured to obtain an identity ID of the TOP1, and obtain identity ID features corresponding to the identity ID from the remaining feature library;
and the second feature matching unit is used for performing similarity calculation on the basis of the first feature information and the second feature information and the ID feature to respectively obtain a first similarity, a second similarity, a third similarity and a fourth similarity.
In the embodiment, when the snapshot is a human face and a front human body, the first row of the similarity matrix is taken as an example, and the top1 similarity value matched in the feature library of the human face registration photo in fig. 2 is referred to.
The specific calculation flow is shown in fig. 3, and mainly includes 3 steps:
the first step is as follows: matching the characteristics of the current face snapshot with a face registry to obtain top1, wherein the similarity 0 is the similarity value between the characteristics of the face to be recognized and the matched top1 ID 1;
the second step is that: respectively acquiring the characteristics corresponding to the ID1 in other 4 characteristic libraries;
the third step: and respectively carrying out similarity calculation on the current face snapshot feature and the front human body snapshot feature and corresponding features to obtain a similarity 1, a similarity 2, a similarity 3 and a similarity 4. By this point, the calculation of the first row in the similarity matrix is finished, and the other rows are similar.
Therefore, the values on the diagonal line from the upper left corner to the lower right corner of the similarity matrix are similarity values of top1 obtained by matching the snapshot features in the corresponding feature library, and the rest values in each row are similarity values of the feature centers of the IDs corresponding to top1 in other libraries and the current to-be-identified corresponding type snapshot features. It should be emphasized that the type of the captured image is particularly emphasized in the above steps because the face captured image only performs similarity calculation with two face feature libraries, and the human body captured image only performs similarity calculation with 3 human body feature libraries.
And the pedestrian re-identification unit is used for matching the identities of the snap-shot personnel based on the similarity matrix so as to output the matched personnel identity IDs.
Specifically, in this embodiment, a corresponding feature library is determined according to the human body angle information in the second feature information, and the person identity ID is determined according to the determined corresponding feature library.
Specifically, in one embodiment, for a human body with a front face, the corresponding feature library is a register _ face registration photo feature library, a capture _ face grabbing photo feature library, or a front _ body front face human body feature library; and determining the personnel identity ID according to the three feature libraries of the register _ face, the capture _ face and the front _ body which correspond to the determination.
In another embodiment: at the current moment, a human face feature A and a back human face feature B exist, firstly, A and a human face registry are used for matching to obtain the similarity D1 corresponding to top1, meanwhile, the ID1 corresponding to top1 is determined, the ID1 is used for finding 4 features C, D, E, F in the rest 4 feature libraries respectively, the similarity D2 is obtained through A and C, and the similarities D3, D4 and D5 are obtained through B and D, E, F respectively. The calculation flows of the other rows in the similarity matrix are similar, a point is clear, and the similarities on the diagonal lines from the upper left corner to the lower right corner are the similarities corresponding to top1 of A and B in the 5 feature libraries.
Identity matching logic:
1. if D1 is greater than the threshold value of the face similarity, the snapshot can be preliminarily judged to belong to ID1, and if D2 and D5 are also greater than the threshold values respectively, the snapshot belongs to ID 1;
2. if D1 is not greater than the threshold, D2 will not be greater than the threshold, and when D5 is greater than the threshold, the identity of the captured picture can also be determined from D5.
The similarity matrix comprehensively reflects the matching conditions of the face to be recognized, the human body and all in-bank IDs, and can conveniently complete the task of re-recognizing pedestrians based on the matrix, and the similarity matrix mainly comprises the following conditions:
1. the snapshot is of human face and human body in front: the method comprises the steps of taking a matching result of a face registration feature library as a main part, taking the similarity of a front human body feature library as correction, and determining the identity of a person to be recognized when two similarity values are greater than respective threshold values; specifically, if the snapshot is a human face and a human body on the front side, in a general case, the similarity of the human face (that is, the value of the 0 position in the similarity matrix) is greater than the threshold, and if the value of the 2 position in the similarity matrix is also greater than the threshold, the identity of the person can be given.
Each position of the similarity matrix in this embodiment has its own threshold.
2. The snapshot is human face and side human body: at the moment, the human face features are influenced by self-shielding, the matching results of the side human body feature library are taken as the main points, and when the similarity is greater than respective threshold values, the identity of the person to be recognized can be determined;
3. the back human body is captured and photographed: when the human face information is lacked, the recognizable features of the human body on the back are less, and as long as the similarity of the human body is greater than a set threshold value, a recognition result is given.
In the above case, the most difficult recognition result is the third one, but the result is equivalent to the effect of the existing back identification method which only depends on the human body characteristics, and in the other two cases, the method of the embodiment is logically better than the scheme which only depends on the human body characteristics.
According to the embodiment, the human face and the multi-angle human body features are fused, so that firstly, the accuracy of identity recognition by simply using the human body features is effectively improved; further, 5 feature libraries are constructed, real-time updating is carried out on the feature libraries, and periodic clustering is carried out to obtain feature centers of each ID in each library; secondly, the real-time updating of 5 feature libraries enables the features closest to the current space-time to be put in storage in time, the feature center obtained by periodic clustering overcomes the problem of the change of the same ID feature under the condition of crossing space-time, and the comprehensive application of the feature center and the feature library ensures the high reliability of identity identification; the embodiment discloses an identity recognition method based on a similarity matrix, which can obviously improve the recognition accuracy. The pedestrian re-identification method of the embodiment is integrated with and applies multi-angle human body features, and is very suitable for actual conditions of various current monitoring scenes, because high-quality front human body or human face images are not acquired under any conditions in the air, and because the human face capture and the human body capture of the same ID are bound, when human body postures do not have identification conditions or human body features of multiple IDs have higher similarity, the identity identification can be still performed by utilizing the human face capture, so that the problem of identity misrecognition and rejection under the conditions of various human body postures or different people with one robe, different people with one robe and the like is effectively solved, and the actual combat effect of pedestrian re-identification is greatly improved.
EXAMPLE III
Referring to fig. 7, fig. 7 is a schematic structural diagram of a pedestrian re-identification device fusing a human face and multi-angle human body features according to the embodiment. The pedestrian re-identification apparatus 20 fusing a human face and multi-angle human body features of the embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. The steps in the above-described method embodiments are implemented when the computer program is executed by the processor 21. Alternatively, the processor 21 implements the functions of the modules/units in the above-described device embodiments when executing the computer program.
Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 22 and executed by the processor 21 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used for describing the execution process of the computer program in the pedestrian re-recognition device 20 for fusing human face and multi-angle human body features. For example, the computer program may be divided into the modules in the second embodiment, and for the specific functions of the modules, reference is made to the working process of the apparatus in the foregoing embodiment, which is not described herein again.
The pedestrian re-recognition device 20 for fusing human faces and multi-angle human features may include, but is not limited to, a processor 21 and a memory 22. It will be understood by those skilled in the art that the schematic diagram is merely an example of the pedestrian re-recognition apparatus 20 fusing the human face and the multi-angle human body features, and does not constitute a limitation of the pedestrian re-recognition apparatus 20 fusing the human face and the multi-angle human body features, and may include more or less components than those shown in the figure, or combine some components, or different components, for example, the pedestrian re-recognition apparatus 20 fusing the human face and the multi-angle human body features may further include an input-output device, a network access device, a bus, and the like.
The Processor 21 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The general processor may be a microprocessor or the processor may be any conventional processor, and the processor 21 is a control center of the pedestrian re-identification device 20 fusing the human face and the multi-angle human body features, and various interfaces and lines are used to connect the various parts of the pedestrian re-identification device 20 fusing the human face and the multi-angle human body features.
The memory 22 may be used to store the computer program and/or module, and the processor 21 implements various functions of the pedestrian re-recognition apparatus 20 that integrates human face and multi-angle human body features by running or executing the computer program and/or module stored in the memory 22 and calling data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, etc. Further, the memory 22 may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Wherein, the module/unit integrated by the pedestrian re-identification device 20 integrating human face and multi-angle human body features can be stored in a computer readable storage medium if it is implemented in the form of software functional unit and sold or used as an independent product. Based on such understanding, all or part of the flow in the method according to the above embodiments may be realized by a computer program, which may be stored in a computer readable storage medium and used by the processor 21 to execute the steps of the above embodiments. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium, etc. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
It should be noted that the above-described device embodiments are merely illustrative, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.
Claims (12)
1. A pedestrian re-identification method fusing human face and multi-angle human body features comprises the following steps:
s1, acquiring a face snapshot image and a human body snapshot image, and extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image to acquire first characteristic information and second characteristic information; the first feature information comprises face feature information and face attribute information, and the second feature information comprises human body feature information and human body attribute information;
s2, comparing the first characteristic information and the second characteristic information which are used as characteristics to be identified with the characteristic centers of 5 preset characteristic libraries to generate a similarity matrix; the 5 feature libraries are a face registration feature library, a face snapshot feature library, a front human body feature library, a side human body feature library and a back human body feature library respectively;
and S3, matching the identity of the candid person based on the similarity matrix to output the matched person identity ID.
2. The method according to claim 1, wherein the generating of the similarity matrix specifically comprises:
s21, obtaining the zeroth similarity of the TOP1 result by the obtained first feature information and a preset first feature library;
s22, acquiring the identity ID of the TOP1, and acquiring the identity ID feature corresponding to the identity ID from the rest feature library;
and S23, performing similarity calculation based on the first characteristic information and the second characteristic information and the ID characteristic, and respectively obtaining a first similarity, a second similarity, a third similarity and a fourth similarity.
3. The method of claim 2, clustering the feature library to obtain the feature centers.
4. The method according to claim 3, further comprising, before the step S2, S11, performing eligibility check on the face snapshot image and the body snapshot image; and extracting the face features and the human body features of the qualified face snapshot image and human body snapshot image to obtain first feature information and second feature information.
5. The method of claim 4, the step S3 comprising: and determining a corresponding characteristic library according to the human body angle information in the second characteristic information, and determining the personnel identity ID according to the determined corresponding characteristic library.
6. The method according to any one of claims 1-4, the body attribute information comprising: the human body angles comprise a front surface, a back surface and side surfaces, and the human body postures comprise a standing posture, a sitting posture and a lying posture; the face attributes include whether the face is wearing a mask.
7. A pedestrian re-recognition device fusing human faces and multi-angle human body features comprises the following units:
the characteristic information acquisition unit is used for acquiring a face snapshot image and a human body snapshot image, extracting face characteristics and human body characteristics of the face snapshot image and the human body snapshot image and acquiring first characteristic information and second characteristic information; the first feature information comprises face feature information and face attribute information, and the second feature information comprises human body feature information and human body attribute information;
the similarity matrix generating unit is used for comparing the captured first characteristic information and the captured second characteristic information serving as characteristics to be identified with a characteristic center of a preset characteristic library to generate a similarity matrix;
and the pedestrian re-identification unit is used for matching the identities of the snap-shot personnel based on the similarity matrix so as to output the matched personnel identity IDs.
8. The apparatus of claim 7, the similarity matrix generating unit further comprising:
the first feature matching unit is used for obtaining the zeroth similarity of the TOP1 result by using the acquired first feature information or second feature information and a preset first feature library;
an identity ID obtaining unit, configured to obtain an identity ID of the TOP1, and obtain identity ID features corresponding to the identity ID from the remaining feature library;
and the second feature matching unit is used for performing similarity calculation on the basis of the first feature information and the second feature information and the ID feature to respectively obtain a first similarity, a second similarity, a third similarity and a fourth similarity.
9. The apparatus of claim 8, clustering the feature degrees to obtain the feature centers.
10. The apparatus according to claim 9, the pedestrian re-recognition unit comprising: and determining a corresponding characteristic library according to the human body angle information in the second characteristic information, and determining the personnel identity ID according to the determined corresponding characteristic library.
11. The apparatus according to any one of claims 7-10, the body attribute information comprising: the human body angles comprise a front surface, a back surface and side surfaces, and the human body postures comprise a standing posture, a sitting posture and a lying posture; the face attributes include whether the face is wearing a mask.
12. A non-volatile memory having stored thereon instructions for, when executed by a processor, implementing the method of pedestrian re-recognition that fuses human faces and multi-angle human features of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460570.3A CN114898402A (en) | 2022-04-28 | 2022-04-28 | Pedestrian re-identification method, device and medium fusing human face and human body characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460570.3A CN114898402A (en) | 2022-04-28 | 2022-04-28 | Pedestrian re-identification method, device and medium fusing human face and human body characteristics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114898402A true CN114898402A (en) | 2022-08-12 |
Family
ID=82719492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210460570.3A Pending CN114898402A (en) | 2022-04-28 | 2022-04-28 | Pedestrian re-identification method, device and medium fusing human face and human body characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114898402A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115862060A (en) * | 2022-11-25 | 2023-03-28 | 天津大学四川创新研究院 | Pig face identification and pig weight identification based pig unique identification method and system |
| CN117058787A (en) * | 2023-08-16 | 2023-11-14 | 鹿客科技(北京)股份有限公司 | Door lock control method, device, electronic equipment and computer readable medium |
-
2022
- 2022-04-28 CN CN202210460570.3A patent/CN114898402A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115862060A (en) * | 2022-11-25 | 2023-03-28 | 天津大学四川创新研究院 | Pig face identification and pig weight identification based pig unique identification method and system |
| CN115862060B (en) * | 2022-11-25 | 2023-09-26 | 天津大学四川创新研究院 | Pig unique identification method and system based on pig face identification and pig re-identification |
| CN117058787A (en) * | 2023-08-16 | 2023-11-14 | 鹿客科技(北京)股份有限公司 | Door lock control method, device, electronic equipment and computer readable medium |
| CN117058787B (en) * | 2023-08-16 | 2024-04-12 | 鹿客科技(北京)股份有限公司 | Door lock control method, device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271554B (en) | Intelligent video identification system and application thereof | |
| Barbosa et al. | Re-identification with rgb-d sensors | |
| JP4610614B2 (en) | Multi-biometric system and method based on a single image | |
| Dagnes et al. | Occlusion detection and restoration techniques for 3D face recognition: a literature review | |
| WO2019242330A1 (en) | Monitoring method, recognition method, related apparatus, and system | |
| CN114898402A (en) | Pedestrian re-identification method, device and medium fusing human face and human body characteristics | |
| JP2021101384A (en) | Image processing apparatus, image processing method and program | |
| CN110443210A (en) | A kind of pedestrian tracting method, device and terminal | |
| JP2006133946A (en) | Moving object recognition device | |
| Kalas | Real time face detection and tracking using OpenCV | |
| CN108416291B (en) | Face detection and recognition method, device and system | |
| CN111241932A (en) | Automobile exhibition room passenger flow detection and analysis system, method and storage medium | |
| KR20170015639A (en) | Personal Identification System And Method By Face Recognition In Digital Image | |
| Denman et al. | Searching for people using semantic soft biometric descriptions | |
| CN105022999A (en) | Man code company real-time acquisition system | |
| CN109271932A (en) | Pedestrian based on color-match recognition methods again | |
| KR102038576B1 (en) | Method of detecting fraud of an iris recognition system | |
| US10915739B2 (en) | Face recognition device, face recognition method, and computer readable storage medium | |
| CN111291612A (en) | Pedestrian re-identification method and device based on multi-person multi-camera tracking | |
| Galiyawala et al. | Person retrieval in surveillance using textual query: a review | |
| CN107862298A (en) | It is a kind of based on the biopsy method blinked under infrared eye | |
| US9286707B1 (en) | Removing transient objects to synthesize an unobstructed image | |
| CN112800923A (en) | Human body image quality detection method and device, electronic equipment and storage medium | |
| WO2023093241A1 (en) | Pedestrian re-identification method and apparatus, and storage medium | |
| CN112418078B (en) | Score modulation method, face recognition device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |