CN104281572B - A kind of target matching method and its system based on mutual information - Google Patents

Abstract

The present invention discloses a mutual information-based target matching method and its system. The method includes: step 1, splicing together features of a query image and a reference image; step 2, matching the spliced feature pairs to The SET feature set under the category, each category corresponds to a SET feature set, and the SET feature set contains the feature pair composed of the query image and the reference image of each category; step 3, use mutual information to characterize the relationship between the SET feature set and its category label Through the calculation of the mutual information, the target matching category is obtained. This method makes full use of the information of multiple pictures in the gallery to improve the matching accuracy and performance.

Description

A Target Matching Method and System Based on Mutual Information

technical field

The invention relates to a multi-category object matching technology in an image database in the field of digital image processing, in particular to an object matching method and system based on mutual information.

Background technique

There are many existing applications for target matching, such as object recognition and classification, image retrieval, etc. For example, in multi-category image database retrieval, for a given query image, score and rank the query image and reference images of each category in the database through the distance measure, and the higher the score ranking is, the final result is obtained. Object matching is generally applied to object retrieval and recognition, and can also be applied to tracking. Here, Person Re-Identification (pedestrian re-examination) is taken as an example to introduce the application of target matching in pedestrian re-examination.

Although computer vision researchers have devoted themselves to the research of human body re-examination in recent years, the problem of human re-examination is still very challenging. This is mainly because of several reasons:

First, in a relatively complex and difficult-to-control camera environment, it is difficult to verify human identity through biological information such as human face or gait.

Secondly, there are many uncertainties in the multi-view of the camera, so it is difficult to obtain robust spatio-temporal information, so the human body re-examination problem is difficult to model through features.

Furthermore, visual appearance features, such as features extracted from human clothing or contours, are relatively direct and not distinguishable; in addition, a person's clothing changes greatly under the condition of multiple cameras, because different cameras Down imaging will be related to lighting, angle, and background occlusion, resulting in different appearances of different people showing different effects under different viewing angles.

Given a query image, in order to find matching reference images from other perspectives, the following two steps are required:

First, first calculate the feature representation of the query image and the database image;

Second, the distance between the two is calculated by a certain distance measure, and then sorted. The top1 obtained is the required result. Sometimes the required result may not be very strict, so the obtained top k can be listed as return candidates results.

Nowadays, the main research methods can be divided into three categories: feature description based methods, metric learning based machine learning methods, and classification based methods. The method based on feature description mainly corresponds to image characteristics and tasks, so as to seek the distinguishability and stability of these feature descriptors for different viewing angles. For example, in document 1 "M.Farenzena, L.B., A.Perina, V.Murino, M.Cristani(2010).Person Re-Identification by Symmetry-Driven Accumulation of Local Features.CVPR.", the unused features are used To describe the same person, first apply the symmetrical method to extract the human skeleton, and then mine the color features of the person’s appearance in the image. This method strongly relies on preprocessing, that is, human body segmentation, so the obtained features are different. will be robust. In document 2 "Bingpeng MA, Y.S., Frederic Jurie(2012).BiCov: a novel image representation for person re-identification and face verification. BMVC." In the document, the HSV three color channels of the image are respectively Gabored, and then the adjacent Feature fusion on the scale of , and finally the covariance descriptor is used to measure the distance between features. For another example, document 3 "Bingpeng Ma, Y.S.(2012).Local Descriptorsencoded by Fisher Vectors for Person Re-identification.Workshop in ECCV." is to abstract the low-level features to the middle-level representation of the so-called Attribute, extract visual or semantic Features, such as for different people, divide them into different clothes in different strips, such as short sleeves, pants, etc.

In contrast, many researchers have started with metric learning, instead of using simple Euclidean distance directly after extracting features, but learning to solve the metric through machine learning methods. Here, the Mahalanobis distance must first be introduced , thus formalizing the human body re-examination as a matching problem, and the metric learning learns M in the above schema. Document 4 "Zheng, W.S. (2012). "Re-identification by Relative Distance Comparison." PAMI." Introduced a method using the minimum correlation distance comparison. The author formalized the idea of using LDA for human re-examination. An optimization optimization problem, and then solve and prove that the whole process is relatively complicated. Document 5 "Martin Kostinger, M.H., Paul Wohlhart, Peter M.Roth, Horst Bischof (2012). Large Scale Metric Learning from Equivalence Constraints. CVPR." The relationship between pairs can be seen from the likelihood ratiotest for derivation, so as to directly solve M, this method is faster and cheaper. Similarly, document 6 "Martin Hirzer, PeterM.Roth, Martin Kostinger, and Horst Bischof(2012).Relaxed Pairwise LearnedMetric for Person Re-identification.ECCV." also used a similar method to write the measure as the F norm to get The final derivation can be obtained by using the trace of the matrix. The advantage of this method is that the calculation amount is small and the speed is fast. In addition, in literature 7 "Prosser, B. (2010). Person Re-Identification by Support VectorRanking.BMVC." also mentioned a kind of similarity and dissimilarity as a binary classification problem, and in literature 8 "[16]Tamar Avraham, I.G., Michael Lindenbaum, and Shaul Markovitch(2012).LearningImplicit Transfer for Person Re-identification.ECCV Workshop."It also mentioned that two feature vectors are directly connected together as a pair, and then use SVM to do two classification problems.

In addition, regarding the task of pedestrian re-examination, the data that can be used for experimental training and testing mainly include three public data sets, VIPER, i-LIDS and ETHZ. ETHZ was originally designed for human detection and tracking, using multi-view cameras to shoot complex street scenes. There are 146 people and 8555 pictures in this data set. In general experiments, the pictures are normalized to a size of 64*128. In the i-LIDS data set collection and complex airports, there are 119 people and 476 pictures, with an average of about 4 pictures per person. In the VIPER dataset, there are a total of 632 people, each with only two pictures.

The same is true for target matching technology. The following summarizes the problems existing in mainstream target matching methods:

A feature-based matching method, which is relatively common, extracts some image features, such as color features, and local features such as SIFT, SURF, or HOG features, and then measures the similarity by Euclidean distance, that is, the closest feature between features Adjacent matching, the matching result can be obtained through a fixed threshold. In addition, the method of using feature matching alone runs slowly, so this feature is later made into a BOW (bag of word) model, that is, features with similar characteristics are quantified to the same cluster center, which speeds up the matching process. Speed, but this method sacrifices accuracy for time. The limitation of the method based on appearance features is that the matching results strongly depend on the feature expression, and the selection of features in different databases is different, which largely depends on the characteristics of the images in the database.

Another mainstream method is the method based on distance metric learning. This method starts from the Mahalanobis distance and uses the machine learning method as a theoretical basis, because the Mahalanobis distance matrix is actually a weighting of different dimensions of data features, or A linear transformation operation of features, and then according to the characteristics of the data, design some objective functions and constraints related to the nature of the data. This constraint enhances the feature dimension with distinguishability, weakens those features that have no discriminative ability, and finally solves Get the Mahalanobis distance matrix. This machine learning method requires a training process during data preprocessing.

Potential problems in research: The method based on low-level feature matching encounters performance bottlenecks, because the visual difference caused by multiple views of the same thing always hinders local feature representation; metric learning methods such as LMNN, ITML, and LDML are quite The complex optimization model is difficult to solve and has high computational complexity. At this stage, effectively combining appropriate features with relatively fast metric learning is a solution to target matching.

Contents of the invention

The object of the present invention is to provide a mutual information-based object matching method and system thereof, which are used to solve the low precision problem existing in the existing object matching technology.

In order to achieve the above object, the present invention provides a target matching method based on mutual information, which is characterized in that it includes:

Step 1, stitch together the features of the query image and the reference image;

Step 2, the spliced feature pairs are corresponding to the SET feature set under the category according to the category composition, each category corresponds to a SET feature set, and the SET feature set includes the feature pair composed of the query image and the reference image of each category;

Step 3, use mutual information to characterize the relationship between the SET feature set and its category label, and obtain the target matching category by processing the mutual information.

The object matching method based on mutual information, wherein, in the step 1, the image feature is a color feature or a texture histogram.

The target matching method based on mutual information, wherein, in the step 3, includes: characterizing the relationship between the SET feature set and its category label with the following formula:

Among them, c represents the value of the category label, N is the maximum value of the category label, S represents the SET feature set corresponding to the c category label, C represents the category label, and MI is the mutual information relationship between the SET feature set and the category label .

The target matching method based on mutual information, wherein, in the step 3, includes: obtaining the target matching category by means of nearest neighbor.

The target matching method based on mutual information, wherein, in the step 3, includes: by sorting the values of the mutual information between all SET feature sets and category labels, the category label corresponding to the maximum value of the mutual information Assigned to the query image to get the target matching category.

In order to achieve the above object, the present invention provides a target matching system based on mutual information, which is characterized in that it includes:

A feature splicing module for splicing together the features of the query image and the reference image;

The category corresponding module is connected to the feature splicing module, which is used to map the spliced feature pairs to the SET feature set under the category according to the category composition, each category corresponds to a SET feature set, and the SET feature set includes the query image and each Feature pairs composed of reference images of categories;

The target matching module is connected to the category corresponding module, and is used to use mutual information to characterize the relationship between the SET feature set and its category label, and obtain the target matching category by processing the mutual information.

In the object matching system based on mutual information, the image feature is a color feature or a texture histogram.

The target matching system based on mutual information, wherein the target matching module characterizes the relationship between the SET feature set and its category label with the following formula:

Among them, c represents the value of the category label, N is the maximum value of the category label, S represents the SET feature set corresponding to the c category label, C represents the category label, and MI is the mutual information relationship between the SET feature set and the category label .

In the target matching system based on mutual information, the target matching module obtains the target matching category through the nearest neighbor method.

The target matching system based on mutual information, wherein the target matching module assigns the category label corresponding to the maximum value of the mutual information to the query by sorting the values of the mutual information between all SET feature sets and category labels image to get the target matching category.

Compared with the prior art, the beneficial technical effect of the present invention is:

The present invention solves the low-precision problem existing in the existing target matching technology, and uses mutual information to represent the relationship between feature pairs and their categories by splicing and pairing pictures, thereby proposing an effective target matching method. This method makes full use of the information of multiple pictures in the gallery to improve the matching accuracy and performance.

First of all, for constructing features, the features extracted from the query image and the reference image are spliced, so as to effectively integrate the information of the query image and the reference image into the pairwise feature; considering making full use of the information of multiple reference images in each category, using these pairwise The feature construction corresponds to the SET feature set of each category, and models the SET feature set and its category label; in order to characterize the relationship between the SET feature set and the category label, the concept of mutual information in information theory is introduced, and each The calculation and sorting of category mutual information, from the perspective of classification, proposes an effective pedestrian re-examination algorithm for robust recognition of monitoring. Experimental results prove that this method greatly improves the performance.

Description of drawings

Fig. 1 is the flow chart of the target matching method based on mutual information of the present invention;

Fig. 2 is a structural diagram of the target matching system based on mutual information in the present invention;

3A-3D are embodiments of target matching based on mutual information in the present invention.

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, but not as a limitation of the present invention. As shown in FIG. 1 , it is a flowchart of the target matching method based on mutual information in the present invention. The process specifically includes the following steps:

Step 101, stitch together the query image and image features in the gallery.

In this step, by splicing the query image and image features in the gallery, the matching feature dimension can be strengthened, and the reference image contains more information.

In image retrieval, one or several images are selected as reference images for each image category, and these image collections form an image database, which is called a gallery.

In step 102, the concatenated feature pairs are mapped to the SET feature set under the category according to the category composition, each category corresponds to a SET feature set, and the SET feature set includes the feature pair composed of the query image and the reference image of each category.

In this step, the SET feature set corresponding to each category is obtained from the pairwise feature, and the query image and the image visual information of the candidate image database are fused together to make full use of the multiple image information in the gallery.

The pairwise feature, that is, the features of the two images are spliced together, called a feature pair. This enhances the ability to express image features. For example, the query image features and the candidate image features in the gallery are spliced together to form a feature pair, which is called pairwise feature.

Step 103, use the mutual information to characterize the relationship between the SET feature set and its category, and finally obtain the target matching category by calculating and sorting the mutual information.

In this step, the mutual information is used to calculate the relationship between the SET feature set and the category, that is, the degree of similarity between the query image and the reference image and the category after splicing. Through the measurement of the similarity, the final category of the picture can be obtained using the nearest neighbor method. , so as to maximize the contribution of multiple reference image information corresponding to different categories.

As shown in FIG. 2 , it is a structural diagram of the target matching system based on mutual information in the present invention. 1, the target matching system 200 includes:

The feature splicing module 201 is used for splicing together the query image and image features in the gallery.

Further, by splicing the query image and image features in the gallery through the feature splicing module 201, the matching feature dimension can be strengthened, and the reference image contains more information.

The category corresponding module 202 and the connection feature splicing module 201 are used to form the spliced feature pairs according to the category into a SET feature set under the corresponding category, each category corresponds to a SET feature set, and the SET feature set includes the query image and each category. Feature pairs composed of reference images.

Further, the category correspondence module 202 obtains the SET feature set corresponding to each category from the pairwise feature, fuses the query image and the image visual information of the candidate image database, and makes full use of the multiple image information in the gallery.

The category correspondence module 202 stitches together the query image and the candidate image features to form a feature pair, thereby enhancing the discrimination ability of image features.

The target matching module 203 and the connection category corresponding module 202 are used to use mutual information to characterize the relationship between the SET feature set and its category, and finally obtain the target matching category by calculating and sorting the mutual information.

Further, the target matching module 203 uses mutual information to calculate the relationship between the SET feature set and the category, that is, the degree of similarity between the query image and the reference image and the category after splicing. Through the measurement of the similarity, the nearest neighbor method can be used to obtain the image The final category, so as to maximize the contribution of multiple reference image information corresponding to different categories.

As shown in Fig. 3A-3D, it is an embodiment of target matching based on mutual information in the present invention. This embodiment is described below with reference to FIGS. 1 and 2 .

In Fig. 3A, a set classification based approach is proposed to solve the image matching technique by utilizing mutual information. As shown in the figure, given a query image, the image category most similar to the query image is searched in the database, thereby obtaining the category label of the query image. In order to overcome this traditional target matching defect, first, the present invention constructs a feature set: for each category, such as categories A, B, and C, there is a SET-based feature set SET A, SET B, and SET C. The feature set includes the set of feature pairs after splicing the query image and all image features in the gallery. The SET feature set represents the query image feature and all images in the gallery to form a new pairwise feature. After the SET feature set is constructed, the spliced features are applied to a SET-CLASS model. The SET-CLASS model is used to represent the relationship between the SET feature set and its corresponding category label. The present invention uses mutual information to represent the relationship between them. In the figure, MI A, MI B, and MIC are used to represent the SET feature set and category label The value of the mutual information between them. Finally, the value of the mutual information is approximated by the nearest neighbor method, and finally the category label corresponding to the maximum value of the mutual information is assigned to the query image by sorting, so as to obtain the category label of the query image

Specific steps are as follows:

S0: Data representation, extracting image features, as shown in Figure 3B.

Given a query image, to measure the similarity between the query image and the images in the database, it is first necessary to extract image features. The present invention extracts the query image and all the image features in the gallery. Since the sizes of the images are different in the actual application scene, it is necessary to unify the image size to the same size before extracting the image features, that is, the image size is normalized. (for example, all images are scaled to 64*128 image resolution). After normalizing the image size, image features are extracted, which can be color features or texture histograms, and different textures in Figure 3B represent different image feature channels. Then, the features of the query image and the image in the gallery are spliced to enhance the feature expression.

S1: Construct a SET feature set, as shown in Figure 3C.

In the gallery, each category, such as pedestrian A, pedestrian B, and pedestrian C (also known as category A, category B, and category C) corresponds to one or more images as reference images. Given a query image, compare the query image The similarity with each category in the gallery, the most similar is judged to belong to this category. As described in step S0, after splicing the query image with the image features of this category in the gallery, that is, the query image feature is x _q , and the image features of category A in the gallery Stitching the two features together constitutes Corresponding to each class, there are multiple spliced feature pairs, that is, for pedestrian B and pedestrian C in the illustration, there are also corresponding spliced feature pairs. In the present invention, the SET feature set includes the splicing of the query image and the image features in the gallery as an assumption, that is, assuming that the query image belongs to this category, the closer the relationship with the category after splicing, the higher the similarity.

S3: Calculate mutual information.

Mutual information is a measure of the interdependence between two sets of variables, a feature set and its class label. For the target matching problem, the present invention can identify the relationship between the SET feature set corresponding to a certain class and the class label by using mutual information:

In the formula, c represents the value of the category label, such as c={1,2,3,...N}, N is the maximum value of the category label, S ^c represents the SET feature set corresponding to the category label c, and C represents the category label , MI is the mutual information relationship between the SET feature set and the category label. Therefore, the final target matching problem becomes a problem of finding the maximum value of mutual information. c represents the value of the category label, such as pedestrian A, pedestrian B, and pedestrian C, and the corresponding category label values c are 1, 2, and 3, respectively. Assuming that all splicing features in each SET feature set are independent of each other, the mutual information expression MI can be written as the following formula, where P represents the probability distribution, and x _qj represents the splicing of the query image q and the image j in the gallery feature pair:

as well as

It can be seen from the above formula that seeking mutual information eventually becomes the solution of the following formula:

In general sense, the probability expression needs to obtain its probability density function, and the calculation is more complicated, so the present invention cites an existing algorithm, and the probability value is approximated by this algorithm, as shown in the following formula

logP(x _qj |C)∝-|x _qj -x ^c || ²

Among them, x ^c represents any feature pair that belongs to the value category c, so the final mutual information expression becomes

where ω(x _qj ) represents NN+ represents the free combination of images of the same type in the gallery, as shown in Figure 3D, |C| represents the number of categories (that is, how many people are in the database), x _qj represents the features of the query image q and the features of the image j in the gallery Feature pair, x _ij represents the feature pair composed of the features of image i and j in the gallery. For x _ij , specifically, NN+ is the target set, a feature pair formed by free combination of features of multiple pedestrian images under the same category. Such a combination It is reflected in the splicing of two images of the same category as the same target feature, and NN- is the free combination of features of two different categories as a non-target feature. In this way, using the nearest neighbor method, the value of mutual information can be approximated.

S4: Nearest neighbor classification.

For each SET feature set, the final result depends on the values of mutual information between all SET feature sets and category labels, sort these values, and then assign the category label with the largest value to the query image, thereby querying the image Get the category labels.

The present invention solves the low-precision problem existing in the existing target matching technology, and uses mutual information to represent the relationship between feature pairs and their category labels by splicing and pairing pictures, thereby proposing an effective target matching method , this method makes full use of the information of multiple pictures in the gallery to improve the matching accuracy and performance.

Of course, the present invention can also have other various embodiments, and those skilled in the art can make various corresponding changes and deformations according to the present invention without departing from the spirit and essence of the present invention. All changes and deformations should belong to the protection scope of the appended claims of the present invention.

Claims (10)

1. A target matching method based on mutual information, characterized in that, comprising: Step 1, stitch together the features of the query image and the reference image; Step 2, the spliced feature pairs are corresponding to the SET feature set under the category according to the category composition, each category corresponds to a SET feature set, and the SET feature set includes the feature pair composed of the query image and the reference image of each category; Step 3, use mutual information to characterize the relationship between the SET feature set and its category label, and obtain the target matching category by processing the mutual information. 2. The object matching method based on mutual information according to claim 1, characterized in that, in the step 1, the image feature is a color feature or a texture histogram. 3. The target matching method based on mutual information according to claim 1 or 2, characterized in that, in the step 3, comprising: characterizing the relationship between the SET feature set and its category label with the following formula: $\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; MI</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>$ Among them, c represents the value of the category label, N is the maximum value of the category label, S represents the SET feature set corresponding to the c category label, C represents the category label, and MI is the mutual information relationship between the SET feature set and the category label . 4. The mutual information-based target matching method according to claim 1 or 2, characterized in that, in step 3, comprising: obtaining the target matching category by means of nearest neighbor. 5. The target matching method based on mutual information according to claim 4, characterized in that, in the step 3, comprising: by sorting the values of the mutual information between all SET feature sets and category labels, the The category label corresponding to the maximum value of mutual information is assigned to the query image to obtain the target matching category. 6. A target matching system based on mutual information, comprising: A feature splicing module for splicing together the features of the query image and the reference image; The category corresponding module is connected to the feature splicing module, which is used to map the spliced feature pairs to the SET feature set under the category according to the category composition, each category corresponds to a SET feature set, and the SET feature set includes the query image and each Feature pairs composed of reference images of categories; The target matching module is connected to the category corresponding module, and is used to use mutual information to characterize the relationship between the SET feature set and its category label, and obtain the target matching category by processing the mutual information. 7. The object matching system based on mutual information according to claim 6, wherein the image feature is a color feature or a texture histogram. 8. The target matching system based on mutual information according to claim 6 or 7, wherein the target matching module characterizes the relationship between the SET feature set and its category label with the following formula: $\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; MI</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>$ Among them, c represents the value of the category label, N is the maximum value of the category label, S represents the SET feature set corresponding to the c category label, C represents the category label, and MI is the mutual information relationship between the SET feature set and the category label . 9. The mutual information-based target matching system according to claim 6 or 7, wherein the target matching module obtains the target matching category by means of nearest neighbor. 10. The target matching system based on mutual information according to claim 9, wherein the target matching module maximizes the mutual information by sorting the values of the mutual information between all SET feature sets and category labels The category label corresponding to the value is assigned to the query image to obtain the target matching category.