WO2019129293A1 - Feature data generation method and apparatus and feature matching method and apparatus - Google Patents

Abstract

Disclosed are a feature data generation method and a feature matching method, comprising: acquiring feature information of an image, and determining a weight corresponding to each type of feature in at least two types of features, wherein a weight corresponding to a first-type feature is greater than weights corresponding to the other features of the at least two types of features, and the first-type feature is a primary feature of acquired features; and according to the weight, weighting the at least two types of features to obtain weighted feature information, storing the weighted feature information in a database so as to generate feature data, or performing feature matching in the database according to the weighted feature information. By means of the embodiments of the present invention, the accuracy of established feature matching can be improved.

Description

Feature data generation and feature matching method and device

This application claims the priority of the Chinese patent application filed on December 29, 2017, the Chinese National Intellectual Property Office, the application number is 201711479219.4, and the application name is “a feature data generation and feature matching method and device”. The citations are incorporated herein by reference.

Technical field

The embodiments of the present invention relate to the field of multimedia technologies, and in particular, to a feature data generation and feature matching method and apparatus.

Background technique

As people pay more and more attention to social public safety, more and more government research institutions and companies are committed to the development of related data content analysis technologies, such as shopping malls, schools, large squares, subway stations and other crowds. In places where public safety incidents have occurred, a large number of surveillance cameras have been deployed to form massive video data. The traditional manual analysis of such huge data has been unable to meet the actual needs. Therefore, the use of computers to realize massive video data The automatic recognition of pedestrians in the field has been rapidly developed.

An important area of computer content analysis technology is feature extraction. Feature extraction mainly implements detection and tracking of targets, and extracts features from regions of interest to characterize source data. The extracted features require a certain commonality to the same type of data, and the heterogeneous data has a higher discrimination, that is, has a stronger discriminating power.

In the existing target recognition problem, the extracted features include color, texture, edge, depth feature, etc. However, in a large data set, it is difficult to obtain a more accurate description of the source data by using a single feature. How to adopt a more concise method, so that the description ability of these features is better enhanced, becomes a problem.

Summary of the invention

Embodiments of the present invention provide a method and apparatus for feature matching and feature information generation, which can improve the accuracy of image feature matching.

In a first aspect, an embodiment of the present invention provides a method for feature matching. The method includes: performing feature extraction on an image to obtain at least two types of features. The features are weighted according to the size of each type of special description ability of the image, wherein the feature weight for the description of the image is larger. Feature information including multiple types of features is obtained according to the weighted features, and feature matching is performed according to the feature information. The ability of a feature to describe an image refers to the degree of discrimination when the image is described according to the feature. By weighting the features according to the description ability of the image, the weight of the features with high description ability in multiple types of special features is enhanced, and the discrimination degree of the weighted feature information in image matching is improved, thereby enhancing the accuracy of image matching. degree.

In one implementation, feature information of an image is acquired, the feature information includes at least two types of features, and the at least two types of features include a first type of feature. Determining a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, and the first feature is the at least The ability to describe the image in the two types of features is greater than the features of the other features in the at least two types of features. And weighting the at least two types of features according to the weights to obtain weighted feature information. Feature matching is performed on the image according to the weighted feature information. By weighting the first type of features, the image description ability of the first type of features is greater than other features, thereby improving the discrimination of the weighted feature information in image matching, and enhancing the accuracy of feature matching.

In one implementation, the description capability of the image is determined according to an indicator value associated with the feature. Correspondingly, the weight corresponding to the first type of feature is determined according to the first indicator value of the first type of feature. For example, when the feature is a texture feature, the ability of the texture feature to describe the image may be determined according to the average amplitude of the image, or a Laplacian; the depth feature may be determined based on the confidence of the feature or by the quality evaluation value. The confidence level is used to describe the possibility that the depth feature is mapped to the corresponding preset interval, and the quality evaluation value is obtained according to the quality evaluation matrix.

In an implementation manner, determining the weight corresponding to the feature according to the indicator value may be according to a formula:

among them,

For the first indicator value of the first type of feature, T ₁ and T ₂ are preset threshold values. when

When the value is between T ₁ and T ₂ , the above formula can be used to determine ω ₁ .

When it is greater than T ₁ , the first type of feature is used as the unique feature of feature matching of the image, that is, ω ₁ takes 1 when

When less than T ₂ , the first type of feature is not used as a feature of feature matching of the image, that is, ω _{1 is} taken as 0.

In an implementation manner, the weight corresponding to the first type of feature is determined according to a feature type of the first type of feature.

In a second aspect, an embodiment of the present invention provides a method for generating feature information, where the method includes: performing feature extraction on an image to obtain at least two types of features. The features are weighted according to the size of each type of special description ability of the image, wherein the feature weight for the description of the image is larger. Feature information including multiple types of features is obtained according to the weighted features, and the weighted feature information is stored in a database. The ability of a feature to describe an image refers to the degree of discrimination when the image is described according to the feature. By weighting the features according to the description ability of the image, the weight of the features with high description ability in multiple types of special features is enhanced, and the discrimination degree of the weighted feature information in image matching is improved, thereby enhancing the accuracy of image matching. degree.

In one implementation, feature information of an image is acquired, the feature information includes at least two types of features, and the at least two types of features include a first type of feature. Determining a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, and the first feature is the at least The ability to describe the image in the two types of features is greater than the features of the other features in the at least two types of features. And weighting the at least two types of features according to the weights to obtain weighted feature information. Feature matching is performed on the image according to the weighted feature information. By weighting the first type of features, the image description ability of the first type of features is greater than other features, thereby improving the discrimination of the weighted feature information in image matching, and enhancing the accuracy of feature matching.

In one implementation, the description capability of the image is determined according to an indicator value associated with the feature. Correspondingly, the weight corresponding to the first type of feature is determined according to the first indicator value of the first type of feature. For example, when the feature is a texture feature, the ability of the texture feature to describe the image may be determined according to the average amplitude of the image, or a Laplacian; the depth feature may be determined based on the confidence of the feature or by the quality evaluation value. The confidence level is used to describe the possibility that the depth feature is mapped to the corresponding preset interval, and the quality evaluation value is obtained according to the quality evaluation matrix.

In an implementation manner, determining the weight corresponding to the feature according to the indicator value may be according to a formula:

among them,

For the first indicator value of the first type of feature, T ₁ and T ₂ are preset threshold values. when

When the value is between T ₁ and T ₂ , the above formula can be used to determine ω ₁ .

When it is greater than T ₁ , the first type of feature is used as the unique feature of feature matching of the image, that is, ω ₁ takes 1 when

When less than T ₂ , the first type of feature is not used as a feature of feature matching of the image, that is, ω _{1 is} taken as 0.

In an implementation manner, the weight corresponding to the first type of feature is determined according to a feature type of the first type of feature.

In a third aspect, an embodiment of the present invention provides an image processing apparatus configured to implement the method and function performed in the above first aspect, implemented by hardware/software, and the hardware/software includes the foregoing functions. The corresponding module.

In a fourth aspect, an embodiment of the present invention provides an image processing apparatus configured to implement the method and function performed in the foregoing second aspect, implemented by hardware/software, and the hardware/software includes the foregoing functions. The corresponding module.

In a fifth aspect, an embodiment of the present application provides an image processing apparatus, including: a processor, a memory, and a communication bus, wherein the communication bus is used to implement connection communication between the processor and the memory, and the processor executes the program stored in the memory. Used to implement the steps in the method of the first aspect above.

In a sixth aspect, an embodiment of the present application provides an image processing apparatus, including: a processor, a memory, and a communication bus, wherein the communication bus is used to implement connection communication between the processor and the memory, and the processor executes the program stored in the memory. Used to implement the steps in the method of the second aspect above.

In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores instructions that, when run on a computer, cause the computer to perform the method of the first aspect or the second aspect.

In an eighth aspect, the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect or the second aspect described above.

In the embodiment of the present invention, the features of the different types are weighted according to the description capability, and the feature matching is performed according to the weighted feature information. Compared with the prior art, the weight of the feature with strong description capability in the feature information is enhanced, thereby enhancing the difference of feature information in feature matching and improving the accuracy of feature matching.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings to be used in the embodiments of the present invention will be described below.

1 is a schematic structural diagram of an image search system according to an embodiment of the present invention;

2 is a schematic flowchart of a feature matching method according to an embodiment of the present invention;

3 is a schematic flowchart of still another feature matching method according to an embodiment of the present invention;

4 is a schematic flowchart of still another feature matching method according to an embodiment of the present invention;

FIG. 5 is a schematic flowchart of a method for generating feature data according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of still another image processing apparatus according to an embodiment of the present invention; FIG.

FIG. 8 is a schematic structural diagram of hardware of an image processing apparatus according to an embodiment of the present invention; FIG.

Detailed ways

The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention.

A feature in the embodiment of the present invention is that a certain type of object is distinguished from corresponding features or characteristics of other types of objects, or a collection of these features and characteristics. The feature is the ability to extract data by measuring or processing. For images, each image has its own features that can be distinguished from other types of images, some are natural features that can be intuitively perceived, such as brightness, edges, textures, and colors; others require transformation or processing. Can be obtained, such as moments, histograms and principal components; and some are depth features obtained by deep learning model extraction.

The ability to describe the same type of feature may be different for different images. For example, for a solid color-based image, since the texture in the image is relatively small, when the texture feature is used to describe the image, it is not possible to distinguish different images with different solid colors. Features When describing such images, it is better to distinguish between different images in such images.

Feature matching refers to an algorithm that performs parameter matching by extracting features of two or more images separately, and then using the described parameters to perform matching. Features processed by feature-based matching typically include features such as color features, texture features, shape features, spatial location features, and the like. Feature matching Firstly, the image is preprocessed to extract its high-level features, and then the matching correspondence between the two images is established. The commonly used feature primitives have some features, edge features and regional features. Feature matching requires many mathematical operations such as matrix operations, gradient solutions, and Fourier transforms and Taylor expansion. Commonly used feature extraction and matching methods are: statistical methods, geometric methods, model methods, signal processing methods, boundary feature methods, Fourier shape description methods, geometric parameter methods, shape invariant moment methods, and so on.

The image in the embodiment of the present invention includes a still image and a moving image.

With reference to FIG. 1, an application scenario of an embodiment of the present invention may be an image search system 1000 based on image feature matching, which may perform on-time analysis and query on a video or an image. The system is composed of four core modules: a feature extraction module 1001, a feature storage module 1002, a feature weighting module 1003, and a feature matching module 1004. The feature extraction module 1001 mainly implements detection and tracking of an image or a video file, and extracts features from the region of interest to obtain feature information data corresponding to the image or video. In addition, the feature extraction module for the target image or video to be retrieved Feature extraction can also be performed, and the extracted features are processed by the feature weighting module. The feature storage module 1002 is configured to construct a database and an index thereof based on the result of feature extraction by the feature module 1001 on the video or the image. The feature weighting module 1003 weights the features extracted in the target image for the target image to be retrieved, and obtains the weighted image feature information. When the database is created, the feature storage module 1002 may establish a database according to the weighted feature obtained by weighting the features extracted by the feature extraction module by the feature weighting module 1003, and obtain the weighted feature database and its index. The feature weighting module 1003 may weight the features of the image or video used to establish the database extracted in the feature extraction module 1001, thereby establishing the weighted result through the feature storage module to establish a weighted database. The image retrieval module 1004 obtains the query result based on the feature information weighted by the target image and matches the image in the database according to the feature matching algorithm. In image matching, the extracted features require certain commonality to the same type of data, and the heterogeneous data has higher discrimination, that is, has stronger discriminating power. However, in a large data set, it is difficult to obtain a more accurate description of the source data using a single feature. Therefore, in the embodiment of the present invention, the extracted feature is weighted by the feature weighting method, so that the finally obtained feature has a stronger description capability, thereby facilitating a better effect by the graph search system.

In some implementations, the image search system 1000 may be a stand-alone computer system, such as a server, to implement the corresponding functions of the feature extraction module 1001, the feature storage module 1002, the feature weighting module 1003, and the feature matching module 1004. In other implementations, image search system 1000 can be a distributed system that includes database nodes and compute nodes for images. The database node stores the database processed by the feature storage module 1002 and its index, and the computing node can implement the corresponding functions of the feature extraction module 1001, the feature weighting module 1002, and the feature matching module 1004. In a distributed system implementation, a module may be deployed on different nodes. For example, the feature extraction module 1001 and the feature weighting module 1003 may be respectively deployed on the database node and the computing node. When generating the feature data, the database node needs to call the feature weighting module 1003 to weight the features extracted by the feature extraction module 1001 to generate a weighted The feature data, while the computing node performs feature matching, also needs to call the feature weighting module 1003 to weight the features of the target image extracted by the feature extraction module 1001 to perform feature matching.

With reference to FIG. 2, it is a flowchart of an image feature matching method according to an embodiment of the present invention. The method includes:

S201. Acquire feature information of the target image.

The target image is the image to be matched in the embodiment of the present invention. The feature information of the target image is extracted by the feature extraction algorithm. In this embodiment, at least two types of different kinds of features need to be extracted, so as to obtain corresponding feature values of each category feature. In a possible implementation, the feature may be a traditional feature, such as a texture feature, a color feature, an edge feature, etc., or may be a depth feature, such as an image feature extracted by a deep learning network. Exemplarily, the extracted color features can be extracted by a color histogram, a color moment, a color set, a color aggregation vector, etc. The extracted texture features can be obtained by a gray level co-occurrence matrix, a Tamura texture feature, an autoregressive texture model, a wavelet transform, and the like. The extracted shape features can be extracted by a boundary feature method, a Fourier shape descriptor method, a geometric parameter method, a shape invariant moment method, and the like.

S202. Set weights on the extracted features, so that the weight of the main features is higher than the weights of the other features extracted. In the embodiment of the present invention, whether a certain feature in the extracted special feature is a main feature is determined by a preset condition. The so-called main feature means that the feature's ability to describe the image is higher than the other features extracted.

In one implementation, the primary feature can be determined by a predetermined category. For example, for a specific target image, the texture feature has a higher ability to describe the image, and the texture feature may be preset as a main feature. When the acquired feature is a texture feature, the feature is considered to be the main feature. For another example, if the head feature in the depth feature of the image is preset as the main feature, when the acquired depth feature is in the head position region of the task in the image, the feature is considered as the main feature.

In another implementation, whether a class of features is a primary feature can be determined by an indicator value that can describe the feature. For example, for a texture feature, whether the texture feature in the image is the main feature can be described by the average amplitude (Dense) of the image and the Laplacian. The average amplitude of the image is obtained by averaging the amplitude of each point in the image. The larger the average amplitude of an image, the stronger the ability of the texture feature of the image to describe the image. When the average amplitude exceeds a predetermined threshold, the texture feature is considered to be the primary feature of the image. As another example, for a depth feature extracted using a VGGM depth model, a corresponding confidence level for the feature can be obtained. The higher the confidence of the feature, the stronger the ability of the feature to describe the image. When the confidence level is higher than the preset threshold, the depth feature corresponding to the confidence is considered to be the main feature.

In yet another implementation, whether the other features are primary features can be determined by describing the index values of the acquired features. For example, when the index value of other features other than one of the collected features is lower than a preset threshold, the feature may be considered as the main feature.

After determining that a type of feature is the main feature, the weight of the feature is set higher than the weight of the other type of feature set by setting weights on the extracted feature. In one implementation, the weights can be set by preset rules. For example, when a feature of a certain type and a category is set as a main feature, a weight corresponding to the category may be preset such that the weight of the feature of the category is higher than the weight of the other type of feature.

In one implementation, the weight of the feature may be adaptively determined based on the index value corresponding to the feature of the category. For example, the empirical threshold of the index value corresponding to the main feature may be set by using the feature as the main feature, and the difference between the index value of the feature and the corresponding empirical threshold is positively correlated with the weight corresponding to the feature, that is, the feature of the feature If the indicator value is greater than the corresponding experience threshold and the difference is larger, the weight corresponding to the feature is larger.

S203. Weight the extracted features according to the determined weights to obtain weighted feature information, and perform feature matching on the image according to the weighted feature information.

The acquired features are weighted to obtain the final weighted feature information. In an implementation manner, after the weighted different kinds of features are serially spliced or the feature dimensions are aligned, the average or maximum value is taken to obtain the final feature information. For example, a 50-dimensional texture feature and a 50-dimensional color feature are extracted, and after weighting the texture feature and the color feature respectively, the final 100-dimensional feature of the image after weighting can be obtained.

After the weighted feature information is obtained, the image can be feature-matched according to the feature information. In different implementations, similarity calculations such as Euclidean distance, Mahalanobis distance, and Hamming distance can be used to characterize the similarity between different features, so as to obtain the final image matching result.

Through the method in this embodiment, different types of features are weighted according to their description capabilities, and feature matching is performed according to the weighted feature information. Compared with the prior art, the weight of the feature with strong description capability in the feature information is enhanced, thereby enhancing the difference of feature information in feature matching and improving the accuracy of feature matching.

FIG. 3 is an image feature matching method according to an embodiment of the present invention. The texture feature and color feature are taken as an example, and the average amplitude is used as an index value for measuring a texture feature or a color feature. The texture features and color features are extracted from the image. The threshold method is used to obtain the weight corresponding to each type of feature. The final feature information of the image is weighted by corresponding weights, thus enhancing the discriminative power of the final feature. It can be understood that for other conventional features, such as edge features, grayscale features, etc., a similar method can be used for matching with reference to this embodiment.

S301. Acquire texture features and color features of the image.

In this embodiment, the target image is a 3-channel color image with a size of 100×100, and the image is extracted from the texture feature and the color feature.

Wherein, the texture features are extracted from the image:

a. Using a 3*3 Sobel operator to extract the gradient dx and dy of each point in the x and y directions;

b. Determine the amplitude of each point, refer to the formula:

Where d _i is the amplitude of a point with coordinates (x _i , y _i ),

with

Gradient in the x and y directions for this point;

c. Delimit the amplitude distribution interval. If the distribution interval is 50 intervals, obtain the distribution histogram of the image amplitude, and obtain the 50-dimensional texture feature corresponding to the image. Reference formula:

among them,

The statistical value of the magnitude falling into the i-th interval.

Extract color features from images:

a. Perform grayscale processing on the target image. The three-channel color value (r, g, b) is weighted to a single-channel gray value Grey, with reference to the formula: Grey=0.299×r+0.587×g+0.114*b

b. Delimit the gray value distribution interval, if the distribution interval is 50 ranges, obtain the distribution histogram of the gray value of the image, thereby obtaining the 50-dimensional color feature corresponding to the image:

among them,

The statistical value of the gray value of the image that falls within the i-th interval.

S302. Determine weights of the texture feature and the color feature according to the average amplitude of the image.

The average amplitude of the image, that is, the amplitude of each point in the image is averaged. Reference formula:

Where d _ij refers to the magnitude of the point of coordinates (i, j), and m and n are the size of the image, which is 100 × 100 in this embodiment.

Corresponding to the thresholds T ₁ and T ₂ (T ₁ > T ₂ ), where T ₁ and T ₂ are threshold values that can be described by the texture image and the color image as unique features when performing feature matching according to experience. Specifically, when the average amplitude is less than T ₂ , the image is a strong color image, and the description feature of the texture feature in the image is weak, and the texture feature may no longer be used as a matching feature when the feature is matched; correspondingly, when the average When the amplitude is greater than T ₁ , the image is a strong texture image, and the color feature description ability in the image is very weak. When the feature is matched, the color feature can no longer be used as a matching feature.

If

Then the image belongs to a strong texture image, ω ₁ =1, ω ₂ =0;

If

Then the image belongs to a strong color image, ω ₁ =0, ω ₂ =1;

If

Then, the weight corresponding to the texture feature and the color feature can be determined according to the formula:

In the embodiment of the present invention, the weight corresponding to the feature is determined by using a double threshold. For texture features, when

The value is greater than

When the texture feature is the main feature in the image, at this time, based on the formula:

Or ω ₁ =1, ω ₂ =0, both will make ω ₁ > ω ₂ , that is, the weight of the texture feature is greater than the color feature. Correspondingly, for color features, when

The value is less than

The color feature is the main feature in the image, at which point ω ₁ <ω ₂ , ie the weight of the color feature is greater than the texture feature.

S303. Adjust weights of the extracted features in the final feature description according to the determined weights, and finally use the weighted texture feature and the serial combination of the color features as the final feature information, that is, obtain the final 100-dimensional feature of the image:

S305. Perform image recognition or retrieval based on feature matching based on the obtained final feature f.

Feature matching can be performed using various feature matching algorithms. For example, the similarity between different features can be characterized by the Euclidean distance to obtain the final image recognition result. Suppose there are N images in the database. Based on the feature weighting form described above, the final obtained features are f _i , i=1, 2,..., N, then the Euclidean distance between the i-th and j-th images is:

The similarity of the two images can be judged according to the Euclidean distance, wherein the smaller the Euclidean distance, the higher the similarity between the two images.

In an embodiment of the invention, the weighting of the color features and texture features of the image is adjusted based on the average magnitude. When the color feature is the main feature of the image, the color feature corresponds to a higher weight, and when the texture feature asks the main feature of the image, the texture feature corresponds to a higher weight. Thereby, the ability of the main features of the image to describe the image is enhanced, thereby enhancing the distinguishability of different features in different images and improving the accuracy of feature matching.

FIG. 4 is still another image feature matching method according to an embodiment of the present invention. The embodiment takes a deep learning feature of a character image as an example. In this embodiment, different depth features are extracted from the image based on the depth model. At the same time, a traditional classifier, such as SVM, or a fully connected layer (fc layer) may be added based on the depth feature to obtain different features. Confidence. Different depth features are weighted according to confidence, thus enhancing the discriminative power of the final depth feature.

In the depth feature learning, for the target image, the depth feature for describing the image can be extracted by the depth model. For depth features, images can be described from different dimensions. For example, the depth feature can describe the possibility that the portrait is male and female in the image from the gender dimension; it can also describe the possibility of the portrait in the image at different ages from the age dimension.

S401. Extract a depth feature from the target image according to the depth model.

Using the VGGM depth model, the depth feature f1 of the human head region is extracted, the feature dimension is n1, and each dimension feature can be characterized as f1 _i , i=1, 2,..., n1; at the same time, the confidence p1 divided into the person is obtained. .

Using the VGGM depth model, the gender feature f2 is extracted, and the feature dimension is n2. Each dimension feature can be characterized as f2 _i , i=1, 2,..., n2; at the same time, the confidence level p2 is obtained. .

Using the VGGM depth model, the age feature f3 of the person is extracted, the feature dimension is n3, and each dimension feature can be characterized as f3 _i , i=1, 2,..., n3; at the same time, the confidence p3 divided into the age segment is obtained. .

Using the VGGM depth model, the depth feature f4 of the clothing style is extracted, the feature dimension is n4, and each dimension feature can be characterized as f4 _i , i=1, 2,..., n4; at the same time, the confidence level classified into the style is obtained. P4.

S402. Adjust the weight of the feature to the pedestrian description according to the confidence level corresponding to each feature, that is, finally obtain the feature of n _d =n1+n2+n3+n4 dimension.

Based on the original depth feature extraction model, a fully connected layer (fc layer) is added to obtain a probability that the feature maps to different classes, for example, a probability that a gender feature of an image is mapped to "male", that is, The confidence pi of different features is obtained. Based on the confidence levels p1, p2, p3, and p4 corresponding to the head, gender, age, and clothing style, the weight corresponding to each feature can be set to ω ₁ = p1,

Thereby the final feature F = concat (ω ₁ × f ₁ , ω ₂ × f ₂ , ω ₃ × f ₃ , ω ₄ × f ₄ ) is obtained.

S403. Perform feature matching based on the obtained features.

The similarity between different features can be characterized by the Euclidean distance to obtain the final image recognition result. For example, there are N images, based on the feature weighting form described above, and the finally obtained features are F _i , i=1, 2, . . . , N, then the Euclidean distance between the i-th image and the j-th image is:

From this distance, the similarity of the two images can be judged.

In one implementation, the final feature can also be determined based on a method of supervised learning. Using traditional supervised learning, such as SVM, the transformation matrix W can be used as the evaluation matrix of the input feature quality. Therefore, for each depth feature fi, the corresponding quality evaluation value pi=W*fi can be obtained, if the dimension of each group feature Different, the dimensionality reduction method such as PCA can be used to align the dimensions of each feature, thus obtaining the final feature F=pooling (p1×f1, p2×f2, p3×f3, p4×f4), wherein pooling can be Take the maximum, minimum or average value.

Referring to FIG. 5, a method for generating image feature data according to the present invention is provided. Through the method in this embodiment, an image feature database for feature matching can be generated. When image feature matching in the foregoing embodiment is performed, feature matching can be performed in the image feature database generated by the method in the method embodiment. Thereby realizing the retrieval function of the image. The method includes:

S501. Acquire feature information of an image to be generated.

The image to be generated, that is, the image in which the feature information is stored in the database is required to be produced in the embodiment of the present invention. The feature information of the image to be generated is extracted by the feature extraction algorithm. In this embodiment, at least two types of different kinds of features need to be extracted, so as to obtain corresponding feature values of each category feature. In a possible implementation, the feature may be a traditional feature, such as a texture feature, a color feature, an edge feature, etc., or may be a depth feature, such as an image feature extracted by a deep learning network. Exemplarily, the extracted color features can be extracted by a color histogram, a color moment, a color set, a color aggregation vector, etc. The extracted texture features can be obtained by a gray level co-occurrence matrix, a Tamura texture feature, an autoregressive texture model, a wavelet transform, and the like. The extracted shape features can be extracted by a boundary feature method, a Fourier shape descriptor method, a geometric parameter method, a shape invariant moment method, and the like.

S502. Set weights on the extracted features, so that the weight of the main features is higher than the weights of other features extracted. In the embodiment of the present invention, whether a certain feature in the extracted special feature is a main feature is determined by a preset condition. The so-called main feature means that the feature's ability to describe the image is higher than the other features extracted.

In one implementation, the primary feature can be determined by a predetermined category. For example, for a specific target image, the texture feature has a higher ability to describe the image, and the texture feature may be preset as a main feature. When the acquired feature is a texture feature, the feature is considered to be the main feature. For another example, if the head feature in the depth feature of the image is preset as the main feature, when the acquired depth feature is in the head position region of the task in the image, the feature is considered as the main feature.

In another implementation, whether a class of features is a primary feature can be determined by an indicator value that can describe the feature. For example, for a texture feature, whether the texture feature in the image is the main feature can be described by the average amplitude (Dense) of the image and the Laplacian. The average amplitude of the image is obtained by averaging the amplitude of each point in the image. The larger the average amplitude of an image, the stronger the ability of the texture feature of the image to describe the image. When the average amplitude exceeds a predetermined threshold, the texture feature is considered to be the primary feature of the image. As another example, for a depth feature extracted using a VGGM depth model, a corresponding confidence level for the feature can be obtained. The higher the confidence of the feature, the stronger the ability of the feature to describe the image. When the confidence level is higher than the preset threshold, the depth feature corresponding to the confidence is considered to be the main feature.

In yet another implementation, whether the other features are primary features can be determined by describing the index values of the acquired features. For example, when the index value of other features other than one of the collected features is lower than a preset threshold, the feature may be considered as the main feature.

After determining that a type of feature is the main feature, the weight of the feature is set higher than the weight of the other type of feature set by setting weights on the extracted feature. In one implementation, the weights can be set by preset rules. For example, when a feature of a certain type and a category is set as a main feature, a weight corresponding to the category may be preset such that the weight of the feature of the category is higher than the weight of the other type of feature.

In an implementation manner, the weight of the feature may be adaptively determined according to the index value corresponding to the category feature. For example, the empirical threshold of the index value corresponding to the main feature may be set by using the feature as the main feature, and the difference between the index value of the feature and the corresponding empirical threshold is positively correlated with the weight corresponding to the feature, that is, the feature of the feature If the indicator value is greater than the corresponding experience threshold and the difference is larger, the weight corresponding to the feature is larger.

S503. Weight the extracted features according to the determined weights to obtain weighted feature information, and store the weighted feature information in the feature database.

The acquired features are weighted to obtain the final weighted feature information. In an implementation manner, after the weighted different kinds of features are serially spliced or the feature dimensions are aligned, the average or maximum value is taken to obtain the final feature information. For example, a 50-dimensional texture feature and a 50-dimensional color feature are extracted, and after weighting the texture feature and the color feature respectively, the final 100-dimensional feature of the image after weighting can be obtained.

After the weighted feature information is obtained, the weighted feature information is stored in the feature database. The feature information in the feature database can be used for feature matching of the image to be retrieved in the foregoing embodiment.

In this embodiment, when feature weighting and weighted feature information are performed, reference may be made to the method of feature weighting and feature information generation in S301, S302, S303 or S401, S402, S403 in the foregoing embodiment. Based on the same inventive concept, the feature weighting method in the aforementioned feature matching corresponds to the feature weighting method in the present embodiment. In the system corresponding to the foregoing FIG. 1 , the method in the foregoing feature matching method is completed by the feature extraction module 1001 , the feature weighting module 1003 , and the feature matching module 1004 . The method in the feature information generation embodiment is performed by the feature extraction module 1001 . The feature storage module 1002 and the feature weighting module 1003 are completed. In the same system, in order to ensure the feature matching and the uniformity of feature information in the feature database, the feature weighting methods corresponding to feature matching and feature information generation should be consistent.

FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention. Referring to the architecture shown in FIG. 1, the apparatus in this embodiment may be a device independent of the feature storage module 1002 and the database in FIG. 1, or may be integrated in the same device as the feature storage module 1002 and the database. The image processing apparatus in this embodiment is configured to perform feature matching in a database after feature extraction and processing of the target image. As shown, the image processing apparatus includes a feature extraction module 601, a feature weighting module 602, and a feature matching module 603. Among them, the description of each module is as follows:

a feature extraction module 601, configured to acquire at least two types of features of the target image;

The feature weighting module 602 is configured to determine weights corresponding to each of the at least two types of features, and weight the at least two types of features according to the set weights to obtain weighted feature information.

The feature matching module 603 performs feature matching on the image according to the weighted feature information.

It should be noted that the implementation of each module may also perform the methods and functions performed in the foregoing embodiments corresponding to the corresponding descriptions of the method embodiments shown in FIG. 2, FIG. 3 or FIG. For example, referring to FIG. 2, the function of the feature extraction module 601 can refer to the method in S201; the function of the feature weighting module 602 can refer to the method in S202, and the method for weighting the feature in S203 to obtain the weighted feature information; the feature matching module 603 can refer to the method of performing image feature matching in S203.

FIG. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention. Referring to the architecture shown in FIG. 1, the device in this embodiment is independent of the device shown in FIG. 6, and may also be a device integrated in the device shown in FIG. 6 to implement the functions of the embodiment. The image processing apparatus in the present embodiment is configured to perform feature extraction on the image data, weight the features, and generate the weighted feature information and store the information in the database. As shown, the image processing apparatus includes a feature extraction module 701, a feature weighting module 702, a feature storage module 703, and a feature information database 704. Among them, the description of each module is as follows:

a feature extraction module 701, configured to acquire at least two types of features of the image;

The feature weighting module 702 is configured to determine weights corresponding to each of the at least two types of features, and weight the at least two types of features according to the set weights to obtain weighted feature information.

The feature storage module 703 is configured to store the weighted feature information in a feature database;

The database 704 is configured to store the feature information weighted by the image data.

It should be noted that the implementation of each module may also perform the methods and functions performed in the foregoing embodiments in accordance with the corresponding description of the method embodiments shown in FIG. 5. At the same time, the feature extraction module 701 and the feature weighting module 702 can also refer to the corresponding feature extraction and feature processing methods of the image to be retrieved in FIG. 2, FIG. 3 and FIG.

Referring to FIG. 8 , it is an apparatus embodiment of an foregoing method embodiment of the present invention. The apparatus may perform the method corresponding to the foregoing FIG. 2, FIG. 3, FIG. 4 or FIG. 5, or may be the foregoing FIG. 6 or 7 A hardware implementation of the device.

The embodiments of the present invention are described with reference to a general computer system environment as an example. As is well known, the device can be adapted to other similar computing hardware architectures to achieve similar functionality. Includes, without limitation, personal computers, service computers, multiprocessor systems, microprocessor based systems, programmable consumer appliances, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices ,and many more.

It can be understood that the embodiments of the present invention can also be implemented by other terminal devices capable of implementing similar computer functions, such as a smart phone, a PAD, a smart wearable device, and the like.

Elements of device 800 may include, but are not limited to, processing unit 820, system memory 830, and system bus 810. The system bus couples various system components including system memory to processing unit 820. System bus 810 can be any of several types of bus structures, which can include a memory bus or memory controller, a peripheral bus, and a local bus using a bus structure. The bus structure may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Extended ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

The heterogeneous processing such as a central processing unit (CPU), a graphics processing unit (Graphics Processing Unit), or a digital signal processing (DSP) processor listed in the foregoing embodiment is included. Device. Wherein, in an embodiment, the central processing unit can be used to perform the method steps in the foregoing embodiments, such as the method steps corresponding to the foregoing FIG. 2, FIG. 3, FIG. 4 or FIG.

Device 800 generally includes a variety of device readable media. The device readable medium can be any medium that can be effectively accessed by any device 800 and includes volatile or nonvolatile media, as well as removable or non-removable media. For example, but not limited to, the device readable medium can comprise a device storage medium and a communication medium. The device storage medium includes volatile and nonvolatile, removable and non-removable media, which can be implemented by any method or technique for storing information such as device readable instructions, data structures, program modules or other data. Device storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, or hard disk storage, solid state hard disk storage, optical disk storage, disk cartridges, disk storage or other storage devices, or any other storage device. The required information and the medium that can be accessed by device 800. Communication media typically includes embedded computer readable instructions, data structures, program modules or other data in a modular data signal (e.g., carrier wave or other transport mechanism) and also includes any medium for information delivery. Combinations of any of the above should also be included within the scope of the device readable media.

System memory 830 includes device storage media, which can be volatile and non-volatile memory, such as read only memory (ROM) 831 and random access memory (RAM) 832. The basic input/output system 833 (BIOS) is typically stored in ROM 831 and contains basic routines that facilitate the transfer of information between elements in device 810. RAM 832 typically contains data and/or program modules that can be accessed and/or operated immediately by processing unit 820. For example, but not limited to, FIG. 8 illustrates operating system 834, application 835, other program modules 836, and program data 837.

Device 800 may also include other removable/non-removable, volatile/non-volatile device storage media. As just one example, FIG. 8 illustrates a hard disk storage 841 which may be a non-removable and non-volatile readable and writable magnetic medium; an external memory 851 which may be a detachable and nonvolatile non-volatile external memory such as Optical disk, magnetic disk, flash memory or mobile hard disk, etc.; hard disk storage 81 is generally connected to system bus 810 through a non-removable storage interface (eg, interface 840), which typically passes through a removable storage interface (eg, interface 860) and system bus 810 is connected.

The drives and their associated device storage media discussed above and illustrated in FIG. 8 provide storage of device readable instructions, data structures, program modules, and other data for device 800. For example, hard drive 841 illustrates storage operating system 842, application 843, other program modules 844, and program data 845. It should be noted that these elements may be the same or different from operating system 834, application 835, other program modules 836, and program data 837.

In this embodiment, the method in the foregoing embodiment or the function of the logic module in the previous embodiment may be read by the processing unit 820 by the code or the readable instruction stored in the device storage medium or may be The instructions are read to perform the method.

When the apparatus performs the method corresponding to the foregoing FIG. 5 or the apparatus corresponding to FIG. 7 in the foregoing embodiment, the aforementioned storage medium, such as the hard disk drive 841 or the external memory 851, may store the feature database in the foregoing embodiment.

The user can enter commands and information through various types of input device 861 devices 800. Various input devices are often connected to the processing unit 820 through a user input interface 860. The user input interface 860 is coupled to the system bus, but can also be connected to the bus structure through other interfaces, such as a parallel interface, or a universal serial. Interface (USB). Display device 890 can also be coupled to system bus 810 via an interface (e.g., video interface 890). Moreover, such as computing device 800 can also include various types of peripheral output devices 820 that can be connected through output interface 880 or the like.

Device 800 can be coupled to one or more computing devices, such as remote computer 870, using logic. A remote computing node includes a device, a computing node, a server, a router, a network PC, an equivalent device, or other general network node, and typically includes many or all of the elements discussed above in connection with device 800. In conjunction with the architecture described above with respect to FIG. 1, the remote computing node can be a slave node, a compute node, or other device. The logical connections illustrated in Figure 8 include a local area network (LAN) and a wide area network (WAN), and may also include other networks. Through logical connections, the device can interact with other nodes in the present invention with other topics. For example, the task information and the data can be transmitted through the logical link with the user, thereby acquiring the task to be assigned by the user; the resource data transmission and the task allocation command are transmitted through the logical link with the computing node, thereby realizing the resources of each node. Acquisition of information and assignment of tasks.

Those skilled in the art will appreciate that in one or more examples described above, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium. Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another. A storage medium may be any available media that can be accessed by a general purpose or special purpose computer.

The specific embodiments of the present invention have been described in detail with reference to the preferred embodiments of the present invention. The scope of the protection, any modifications, equivalent substitutions, improvements, etc., which are made on the basis of the technical solutions of the present invention, are included in the scope of the present invention.

.

Claims (22)

1. A method of feature matching, characterized in that the method comprises

   Obtaining feature information of the image, where the feature information includes at least two types of features, and the at least two types of features include the first type of features;

   Determining a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, and the first feature is the at least The ability to describe the image in the two types of features is greater than the features of the other features in the at least two types of features;

   And weighting the at least two types of features according to the weights to obtain weighted feature information;

   Feature matching is performed on the image according to the weighted feature information.
2. The method according to claim 1, wherein the weight corresponding to the first type of feature is determined according to a first indicator value of the first type of feature, and the first indicator value is used to indicate that the first feature is The ability to describe the image.
3. The method of claim 2 wherein said first type of feature is a texture feature and said first indicator value is an average magnitude of the image.
4. The method according to claim 2 or 3, wherein determining the weight corresponding to each of the at least two types of features comprises:

   

   For the first indicator value of the first type of feature, T ₁ and T ₂ are preset thresholds, when

   

   Determining the weight of the first type of feature when greater than or equal to T ₂ and less than or equal to T ₁

   

   when

   

   When it is greater than T ₁ , ω ₁ =1, when

   

   When it is less than T ₂ , ω ₁ =0.
5. The method according to claim 1 or 2, wherein the feature is a depth feature, and the determining the weight corresponding to each of the at least two types of features comprises:

   A corresponding weight is determined according to a confidence level of each type of feature, wherein the confidence is used to describe a possibility that the depth feature is mapped to a corresponding preset interval.
6. The method according to claim 1, wherein the weight corresponding to the first type of feature is determined according to a feature type of the first type of feature.
7. A method for generating feature data, the method comprising:

   Obtaining feature information of the image, where the feature information includes at least two types of features, and the at least two types of features include the first type of features;

   Determining a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, and the first feature is the at least The ability to describe the image in the two types of features is greater than the features of the other features in the at least two types of features;

   And weighting the at least two types of features according to the weights to obtain weighted feature information;

   The weighted feature information is stored in a feature database.
8. The method according to claim 7, wherein the weight corresponding to the first type of feature is determined according to a first indicator value of the first type of feature, and the first indicator value is used to indicate that the first feature is The ability to describe the image.
9. The method according to claim 8, wherein the first type of feature is a texture feature, and the first indicator value is an average amplitude of the image;
10. The method according to claim 8 or 9, wherein determining the weight corresponding to each of the at least two types of features comprises:

    

    For the first indicator value of the first type of feature, T ₁ and T ₂ are preset thresholds, when

    

    Determining the weight of the first type of feature when greater than or equal to T ₂ and less than or equal to T ₁

    

    when

    

    When it is greater than T ₁ , ω ₁ =1, when

    

    When it is less than T ₂ , ω ₁ =0.
11. The method according to claim 7 or 8, wherein the feature is a depth feature, and the determining the weight corresponding to each of the at least two types of features comprises:

    A corresponding weight is determined according to a confidence level of each type of feature, wherein the confidence is used to describe a possibility that the depth feature is mapped to a corresponding preset interval.
12. The method according to claim 7, wherein the weight corresponding to the first type of feature is determined according to a feature type of the first type of feature.
13. A feature matching device, characterized in that the device comprises:

    a feature extraction module, configured to acquire feature information of an image, where the feature information includes at least two types of features, and the at least two types of features include a first type of feature;

    a feature weighting module, configured to determine a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, the first a feature that is characterized in that the image description capability of the at least two types of features is greater than other features of the at least two types of features; and weighting the at least two types of features according to the weights to obtain a weighted Characteristic information

    And a feature matching module, configured to perform feature matching on the image according to the weighted feature information.
14. The device according to claim 13, wherein the weight corresponding to the first type of feature is determined according to a first indicator value of the first type of feature, and the first indicator value is used to indicate that the first feature is The ability to describe the image.
15. The device according to claim 14, wherein the feature weighting module determines that the weight corresponding to each of the at least two types of features comprises:

    

    For the first indicator value of the first type of feature, T ₁ and T ₂ are preset thresholds, when

    

    Determining the weight of the first type of feature when greater than or equal to T ₂ and less than or equal to T ₁

    

    when

    

    When it is greater than T ₁ , ω ₁ =1, when

    

    When it is less than T ₂ , ω ₁ =0.
16. A feature data generating device, characterized in that the device comprises:

    a feature extraction module, configured to acquire feature information of an image, where the feature information includes at least two types of features, and the at least two types of features include a first type of feature;

    a feature weighting module, configured to determine a weight corresponding to each of the at least two types of features, wherein the weight of the first type of feature corresponds to a weight corresponding to other features of the at least two types of features, the first a feature that is characterized in that the image description capability of the at least two types of features is greater than other features of the at least two types of features; and weighting the at least two types of features according to the weights to obtain a weighted Characteristic information

    And a feature storage module, configured to store the weighted feature information into a feature database.
17. The device according to claim 16, wherein the weight corresponding to the first type of feature is determined according to a first indicator value of the first type of feature, and the first indicator value is used to indicate that the first feature is The ability to describe the image.
18. The device according to claim 17, wherein

    The feature weighting module determines that the weights corresponding to each of the at least two types of features include:

    

    For the first indicator value of the first type of feature, T ₁ and T ₂ are preset thresholds, when

    

    Determining the weight of the first type of feature when greater than or equal to T ₂ and less than or equal to T ₁

    

    when

    

    When it is greater than T ₁ , ω ₁ =1, when

    

    When it is less than T ₂ , ω ₁ =0.
19. A computer system, comprising: at least one processor and at least one memory, wherein

    The memory stores computer program instructions, the processor reading the computer program instructions to perform the method of any of claims 1-6.
20. A computer system, comprising: at least one processor and at least one memory, wherein

    The memory stores computer program instructions, the processor reading the computer program instructions to perform the method of any of claims 7-12.
21. A computer readable storage medium, characterized in that the computer readable storage medium comprises instructions for performing the operation of any one of claims 1-6 when the instructions are run on a computer Methods.
22. A computer readable storage medium, characterized in that the computer readable storage medium comprises instructions for performing the operation of any one of claims 7-12 when the instructions are run on a computer Methods.

Images