CN111353062A - Image retrieval method, device and equipment - Google Patents
Image retrieval method, device and equipment Download PDFInfo
- Publication number
- CN111353062A CN111353062A CN201811568022.2A CN201811568022A CN111353062A CN 111353062 A CN111353062 A CN 111353062A CN 201811568022 A CN201811568022 A CN 201811568022A CN 111353062 A CN111353062 A CN 111353062A
- Authority
- CN
- China
- Prior art keywords
- matching
- image
- query image
- candidate set
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application discloses an image retrieval method, which comprises the steps of firstly carrying out first-stage matching on a query image input by a user and a hash value of global features of a matched image in a matched image library to obtain a first candidate set comprising a rough matching result. Then, the global features of the matching images in the first candidate set and the global features of the query image are subjected to second-stage matching, and a second candidate set comprising fine matching results is obtained. And finally, performing third-stage matching on the local features of the matched images in the second candidate set and the local features of the query image to obtain a final retrieval result. The method adopts a three-stage retrieval method to obtain the matching images similar to the query image, wherein the coarser the matching process is, the faster the matching speed is, the more the filtered matching images are, the more the matching of various features is required to be repeated for each matching image, and the image retrieval speed is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to an image retrieval method, and a corresponding apparatus and device.
Background
Image retrieval is a technique of retrieving a matching image similar or identical to a query image input by a user. The technology has wide application fields, such as search engines, infringement image recognition and the like. Taking infringement image identification as an example, in the current copyright market, a large number of copyright images are randomly used without authorization, so technical means are required to be utilized to retrieve the infringement images in the network, and image organizations and photographers are helped to further attack the infringement pirate images. The data volume of the current network image is huge, and higher requirements are put forward on the efficiency of image retrieval.
Disclosure of Invention
The application provides an image retrieval method which improves the efficiency of image retrieval.
In a first aspect, an image retrieval method is provided, executed by at least one computing device, for retrieving a matching image that is the same as or similar to a query image input by a user, the method comprising the following three-segment matching process:
matching the first section: acquiring a hash value of the global feature of a query image, matching the hash value of the global feature of the query image with the hash value of the global feature of a matching image library; and selecting a first candidate set from the matching images in the matching image library according to the matching result of the hash value of the global feature of the query image and the hash value of the global feature of the matching images in the matching image library.
And second section matching: acquiring global features of the query image, and matching the global features of the query image with the global features of the matched images in the first candidate set; and selecting a second candidate set from the matching images of the first candidate set according to the matching result of the global features of the query image and the global features of the matching images in the first candidate set.
And matching the third section: obtaining local features of the query image, and matching the local features of the query image with the local features of the matched images in the second candidate set; and determining a retrieval result from the matching images in the second candidate set according to the local features of the query image and the matching results of the local features of the matching images in the second candidate set. The hash value, the global feature and the local feature of the global feature of the matched image library are extracted in advance and stored in the feature library before the method is executed.
The first stage of matching is realized through the matching of the hash value, a large number of matched images are rapidly filtered, and the image retrieval efficiency is improved. The second section matching and the third section matching further finely match the filtered matching images, and the accuracy of image retrieval is ensured.
In one possible implementation, the method further includes: acquiring the position information of the local feature of the query image, wherein the position information of the local feature of the query image comprises the coordinate information of the local feature of the query image or the area information of the local feature of the query image in the query image;
when the position information of the local feature of the query image includes the coordinate information of the local feature of the query image, acquiring region information of the local feature of the query image according to the coordinate information of the local feature of the query image;
the local features of the query image are obtained and matched with the local features of the query image and the local features of the matched images in the second candidate set; determining a retrieval result from the matching images of the second candidate set according to the matching result of the local feature of the query image and the local feature of the matching images of the second candidate set comprises:
matching region information of local features of the query image with region information of local features of matching images in the second candidate set; and determining the retrieval result from the matching images of the second candidate set according to the matching result of the region information of the local feature of the query image and the region information of the local feature of the matching images in the second candidate set, wherein the similarity between the region information of the local feature of the matching images in the retrieval result and the region information of the local feature of the query image is greater than a first threshold value.
The position information of the local features is adopted to screen the matched images as the retrieval result, so that the wrong retrieval result caused by the mismatching of the local features can be reduced, and the retrieval precision is further improved.
In one possible implementation, the method further includes: acquiring the position information of the local features of the query image, wherein the position information of the local features of the query image comprises the area information of the local features of the query image in the query image;
when the position information of the local feature of the query image includes the coordinate information of the local feature of the query image, acquiring region information of the local feature of the query image according to the coordinate information of the local feature of the query image;
the local features of the query image are obtained and matched with the local features of the query image and the local features of the matched images in the second candidate set; determining a retrieval result from the matching images of the second candidate set according to the matching result of the local feature of the query image and the local feature of the matching images of the second candidate set comprises:
matching local features of the query image with local features of matching images in the second candidate set; acquiring a first region distribution vector of the local features of the matched query image in the query image according to the region information of the local features of the query image; acquiring a second region distribution vector of the local features of the matched images of the second candidate set in the matched images of the second candidate set according to the region information of the local features of the matched images of the second candidate set; and judging whether the similarity of the first region distribution vector and the second region distribution vector is greater than a second threshold value, and if the similarity of the first region distribution vector and the second region distribution vector is greater than the second threshold value, determining that the matched image in the second candidate set is included in the retrieval result.
The matched local features are regressed to the query image and the matched image to obtain the regional information of the query image and the matched image, and the regional information of the matched local features in the query image and the regional information of the matched local features in the matched image are compared, so that the negative influence of the matched local features on the retrieval result is further reduced, and the retrieval precision is improved.
In a second aspect, an image retrieval apparatus is provided, including: the device comprises a first matching module, a second matching module and a third matching module. The three modules may be implemented by software running on a computing device, and the three modules may run on different computing devices.
In a third aspect, a computing device system is provided, which includes at least one computing device, each computing device including a processor and a memory, the processor of the at least one computing device being configured to execute program code in the memory to perform the method provided in the first aspect or any one of the possible implementations of the first aspect.
In a fourth aspect, a non-transitory readable storage medium is provided, which when executed by at least one computing device, performs the method provided in the first aspect or any one of the possible implementations of the first aspect. The storage medium stores a program therein. Types of the storage medium include, but are not limited to, volatile memory, such as random access memory, non-volatile memory, such as flash memory, hard disk (HDD), and Solid State Drive (SSD).
A fifth aspect provides a computing device program product which, when executed by at least one computing device, performs the method provided in the preceding first aspect or any possible implementation manner of the first aspect. The computer program product may be a software installation package, which may be downloaded and executed on a computing device in case it is desired to use the method as provided in the first aspect or any one of the possible implementations of the first aspect.
In the sixth aspect, an image retrieval method is further provided, in the process of matching the local features of the query image with the local features of the matching image, the local features of the query image and the local information of the local features of the matching image are represented by a region distribution vector, and whether the query image is matched with the matching image is judged by comparing the similarity between the region distribution vectors. The method reduces the negative influence of the local characteristics of the matching errors on the retrieval result, and improves the retrieval precision. The method comprises the following steps:
acquiring position information of at least one local feature of a query image and position information of at least one local feature of a matching image, wherein the position information of each local feature of the query image comprises area information of the local feature of the query image in the query image, and the position information of each local feature of the matching image comprises area information of the local feature of the matching image in the matching image;
when the position information of the local feature of the query image includes the coordinate information of the local feature of the query image, acquiring region information of the local feature of the query image according to the coordinate information of the local feature of the query image;
acquiring a first region distribution vector of at least one local feature of the query image in the query image according to region information of the at least one local feature of the query image, wherein the first region distribution vector comprises the number of local features included in each region of the query image;
acquiring a second region distribution vector of at least one local feature of the matched image in the matched image according to region information of the at least one local feature of the matched image, wherein the second region distribution vector comprises the number of local features included in each region of the matched image;
determining that a similarity of the first region distribution vector and the second region distribution vector is greater than a threshold;
determining that the query image matches the matching image.
In a seventh aspect, there is also provided a computing device system comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device being configured to execute program code in the memory to perform the method provided in the sixth aspect.
In an eighth aspect, a non-transitory readable storage medium is provided, which when executed by at least one computing device, performs the method provided by the foregoing sixth aspect.
A ninth aspect. There is also provided a computing device program product which, when executed by at least one computing device, performs the method provided by the preceding sixth aspect.
Drawings
In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly described below.
FIG. 1 is a schematic diagram of a system architecture provided herein;
FIG. 2 is a schematic diagram of a system operating environment provided by the present application;
FIG. 3 is a schematic diagram of yet another system operating environment provided by the present application;
FIG. 4 is a schematic diagram of yet another system operating environment provided herein;
FIG. 5 is a flow chart of a pre-processing method provided herein;
fig. 6 is a schematic diagram of position information of feature points provided in the present application;
FIG. 7 is a schematic diagram of region partitioning of feature points provided herein;
FIG. 8 is a flow chart of an online processing method provided herein;
FIG. 9 is a schematic illustration of location information provided herein;
FIG. 10 is a schematic structural diagram of a pretreatment device provided herein;
fig. 11 is a schematic structural diagram of an image retrieval apparatus provided in the present application;
fig. 12 is a schematic structural diagram of an image retrieval system provided in the present application.
Detailed Description
The technical method in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the present application, there is no logical or temporal dependency relationship between the respective "first", "second", and "nth".
Global features, features extracted from an image (image) as a whole, are generally used to express the overall characteristics of the image. Global features include any one or a combination of more of the following: pre-training deep learning features, GIST features, Local Binary Pattern (LBP) features, or color distribution features, etc. The pre-training deep learning features are image features obtained by calculation by using a pre-training convolutional neural network model, and the pre-training model is a model obtained by image training in a data set. A data set, such as the ImageNet project, is a large visual database for visual object recognition software research, containing 1000 categories of tens of millions of images. After the image is input into the convolutional neural network model trained in advance, the features in any convolutional layer of the model can be used as the global features of the image. The color distribution characteristics refer to image characteristics represented by color statistical distribution vectors, such as color histograms, color moments, color aggregation vectors and the like; the color histogram refers to distribution characteristics of an image color space in different color intervals, such as a Red Green Blue (RGB) color distribution space; the color moments include first order moments, second order moments, third order moments, and the like. GIST features are an abstract scene representation that can naturally motivate the notion of different categorical scenes, such as cities, mountains, etc. GIST features were originally published in Friedman A. framework graphics: The role of knowledge in automated encoding and management for The field [ J ]. Journal of experimental pathology: General,1979,108(3): 316.
The local feature is a feature extracted by using a partial region of an image as an input, and is generally used to express a local characteristic of an image. The local features include any one or a combination of more of the following: scale Invariant Feature (SIFT) features, accelerated up robust features (SURF), affine (affine) SIFT features (also called ASIFT features), principal component analysis-based scale invariant features (PCA-SIFT), Histogram of Oriented Gradients (HOG), oriented FAST and rotated BRIEF descriptor features (ORB) where features of accelerated segmentation detection (features) are, binary robust invariant scale features (FAST and robust features), binary robust features (secure, binary, or invariant features), etc. Local features are generally extracted centering on a certain pixel in the image, which is also called an interest point, a key point or a feature point. Each type of descriptor extracted in the local range of the feature point generally has a feature with a constant local scale, that is, if the content in the local range of the feature point is subjected to various types of processing, such as scaling, rotation, noise addition, shielding, and illumination change, the descriptor extracted in the local range after the processing is the same as or similar to the descriptor extracted in the local range before the processing. Therefore, if there are a large number of local feature matches, i.e. there are a large number of feature point matches, it can be generally considered that the similarity between the two images is high. A plurality of local features can be generally extracted for each type of local feature in one image.
Similarity, which refers to the degree of similarity between two hash values, features, or vectors. The similarity may be a distance between two hash values, features or vectors, or a value obtained by normalizing the distance between two hash values, features or vectors. The distance here may be a euclidean distance, a cosine distance, an edit (editor) distance, a hamming distance, etc. The smaller the distance between two hash values, features or vectors, the greater the similarity.
Fig. 1 illustrates a block diagram of a system provided by the present application, which is divided into a preprocessing portion 100 and an online processing portion 200. The preprocessing portion 100 is used to extract the corresponding hash value, global feature and local feature and their location information for the matching images in the matching image library 105. The online processing section 200 is configured to retrieve similar images for the query image input by the user within the matching image library 105 and provide the retrieval result to the user. Since the number of matching images stored in the matching image library 105 may be huge, the preprocessing section 100 generally needs to be preprocessed before the online processing section 200 works. The preprocessing portion 100 can periodically extract the corresponding hash value, global feature and local feature and their location information for the newly added matching image in the matching image library to maintain the real-time performance of the search result of the query image. The on-line processing part 200 and the preprocessing part 100 can also process in parallel to further improve the working efficiency of the system.
The matching image library 105 stores matching images, and each matching image is input to the local feature extraction module 101 and the global feature extraction module 102, respectively. The local feature extraction module 101 extracts the local features of the matching image and the position information thereof, and stores the local features and the position information thereof in the feature library 104. The global feature extraction module 102 extracts global features of the matching images and stores the global features into the feature library 104. The global feature extraction module 102 further inputs the extracted global features into the hash module 103, and the hash module 103 hashes the input global features to obtain hash values corresponding to the global features, and stores the obtained hash values into the feature library 104.
Query images input by a user through the query interface 207 are input to the local feature extraction module 201 and the global feature extraction module 202, respectively. The global feature extraction module 202 extracts global features of the query image and inputs the global features to the hashing module 203 and the second matching module 205. The hash module 203 generates a hash value from the global features input from the global feature extraction module 202, and inputs the generated hash value to the first matching module 204. The first matching module 204 matches the hash value of the matching image 106 in the feature library 104 with the hash value generated by the hashing module 203, and selects a first candidate set from the matching image library 105 according to the matching result. The similarity of the hash value of the matching image within the first candidate set to the hash value of the query image is above a threshold. The matching images in the first candidate set are input into the second matching module 205, the second matching module 205 matches the global features of the matching images in the first candidate set with the global features generated by the global feature extraction module 202, and a part or all of the matching images in the first candidate set are selected as the second candidate set according to the matching result. The global features of the matching images within the second candidate set have a similarity to the global features of the query image above a threshold. The images in the second candidate set are input to a third matching module 206. The local feature extraction module 201 extracts the local feature of the query image and the position information thereof, and inputs the local feature and the position information thereof to the third matching module 206. The third matching module matches the local features and the position information of the images in the second candidate set with the local features and the position information of the local features generated by the local feature extraction module 201, and selects part or all of the matched images from the images in the second candidate set as search results according to the matching results and returns the search results to the user through the query interface/interface 207.
Optionally, the local feature extraction module 201 of the online processing part is the same as the local feature extraction module 201 of the preprocessing part. The global feature extraction module 202 of the online processing portion is the same as the global feature extraction module 202 of the preprocessing portion. The hash module 203 of the online processing portion is the same as the hash module 103 of the preprocessing portion.
The system shown in fig. 1 includes at least the following two operating environments. As shown in fig. 2, both the preprocessing portion 100 and the online processing portion 200 are deployed on the cloud. The system provides image retrieval functionality to the user in the form of a cloud service, and the user transmits a query image to the cloud via a query interface 207 (application programming interface or browser). And after a matching image which is the same as or similar to the query image is searched on the cloud, returning the retrieval result to the user.
As shown in fig. 3, the preprocessing portion 100 is deployed on the cloud, and performs feature extraction on the huge matching image library 105 by using abundant computing resources and storage resources on the cloud. The online processing part 200 is deployed on a user device, and the user device performs the functions of extracting and matching the features of the query image. Since the online processing portion 200 requires less computing resources and storage resources, placing the online processing portion 200 on the user device in the runtime environment shown in fig. 3 further makes reasonable use of the resources of the cloud and the user device. In the operating environment shown in fig. 3, the online processing section 200 may be provided to the user by the operator of the cloud in the form of a software module and installed on the user device.
As shown in fig. 4, the preprocessing portion 100 is deployed on the cloud, and the first matching module 204, the second matching module 205, and the third matching module 206 are also deployed on the cloud. Other modules of the online processing section 200 are deployed on the user device. In the operating environment shown in fig. 4, the first matching module 204, the second matching module 205, the third matching module 206, and the feature library 104 are all deployed on the cloud, so that network resource consumption caused by transmission of hash values, global features, and local features between the feature library 104 and the first matching module 204, the second matching module 205, and the third matching module 206 is avoided. Meanwhile, the rest modules of the online processing part 200 are placed on the user equipment, so that resources of the cloud and the user equipment are further reasonably utilized. In the operating environment shown in fig. 4, the local feature extraction module 201, the global feature extraction module 202, the hash module 203, and the query interface/interface 207 may be provided to the user by the cloud operator in the form of software modules and installed on the user device.
Fig. 5 provides a flow chart of a preprocessing method performed by the preprocessing portion 100.
S301, the local feature extraction module 101 obtains the matching image and the identification ID of the matching image from the matching image library 105. Each matching image stored in the matching image library 105 has a unique ID.
S302, the local feature extraction module 101 extracts local features for the matching image, and sends the extracted local features, corresponding position information and matching image ID to the feature library 104.
The local feature extraction module 101 may comprise at least one sub-module, each for extracting a local feature, for example, the SIFT sub-module for extracting SIFT local features from the matching image and the SURF sub-module for extracting SURF local features from the matching image. The extracted local features are sent to the feature library 104 along with the matching image ID. The position information of the feature point of each local feature, such as coordinates or area information within the matching image, is also sent to the feature library 104 as the position information of the local feature.
S303, the matching image library 105 sends the matching image and the identification ID of the matching image to the global feature extraction module 102.
S304/S306, the global feature extraction module 102 extracts global features for the matching images and sends the extracted global features and the matching image IDs to the hash module 103 and the feature library 104.
The global feature extraction module 102 may comprise at least one sub-module, each sub-module for extracting a local feature, for example, an RGB sub-module for extracting RGB global features from the matching image and an LBP sub-module for extracting LBP global features from the matching image. The extracted global features are sent to the feature library 104 along with the matching image ID. After the global feature extraction module 102 extracts the global features, S304 and S306 may be executed in parallel or S304 and S306 may be executed in any sequential order.
S305, the hash module 103 hashes the global feature to generate a hash value, and sends the generated hash value and the matching image ID to the feature library 104.
S301-S302 and S303-S306 may be performed in parallel or in any sequential order.
After S301-S306 are executed for each matching image in the matching image library 105, a series of information of each matching image, including ID, hash value, global feature, local feature and position information thereof, is stored in the feature library 104, as shown in table 1:
| image ID | Hash value | Global features | Local features | |
| ID | ||||
| 1 | |
|
Local feature 1-1 | Location information 1-1 |
| Global feature 2 | Local features 1-2 | Location information 1-2 | ||
| Global feature 3 | ... | ... | ||
| ... | Local feature 2-1 | Location information 2-1 | ||
| Global feature M | ... | ... | ||
| Local feature 3-1 | Location information 3-1 | |||
| ... | ... | |||
| Local feature N-1 | Position information N-1 | |||
| ID 2 | ... | ... | ... | ... |
TABLE 1
Table 1 records global feature 1 through global feature M extracted from the matching image ID1, where each global feature refers to a global feature, for example, global feature 1 may be an RBG global feature, and global feature 2 may be an LBP global feature. Table 1 also records hash value 1 generated by hashing global feature 1 to global feature M, where hash value 1 may be composed of at least one hash value, that is, global feature 1 to global feature M may be grouped togetherThe hash value 1 is obtained after the combination, or the global feature 1 to the global feature M may be divided into a plurality of segments and then the segments are respectively hashed, and a plurality of hash values are hashed to form the hash value 1. The global features matching the image ID1 include M, each global feature including MiFor example, 1 ≦ i ≦ M, the hash value 1 of the matching image ID1 may be set to ∑ miThe M global features can be randomly combined, and the combined result is hashed to obtain a hash value, in this case, the hash value 1 is less than ∑ MiA hash value.
Illustratively, the hashing algorithm employed by the hashing module 103 may be such that positive numbers are mapped to 1 and non-positive numbers are mapped to 0. Using this hashing algorithm and each dimension of each global feature is hashed to a hash value, a total of ∑ m of image ID1 is matchediThe global feature of each dimension is mapped to ∑ miThe binary sequence of the bit greatly reduces the complexity of the global features and improves the matching speed of the subsequent first candidate set.
Table 1 also records local features 1-1 to local feature N-1 extracted from the matching image ID1 and position information of each local feature. Since a plurality of feature points can be generally extracted for each local feature in the matching image ID1, for example, the local features 1-1 and 1-2 are SIFT local features, and the local feature 201 is a SURF local feature. In table 1, a total of M global features and N local features are recorded, and M and N are positive integers.
Fig. 6 provides a schematic illustration of the location information for each local feature recorded in the feature library 104. The image in fig. 6 includes 10 feature points, each feature point corresponds to a local feature, and different feature points may correspond to the same or different kinds of local features. The position information of the local feature corresponding to each feature point includes coordinate information of the feature point in the image or area information of the feature point in the image. Each image is divided into H regions, H being a positive integer greater than 1. As shown in fig. 7, the query image and the matching image may each be divided into n rows and m columns of regions, where n × m equals H, and m and n are positive integers, e.g., the image in fig. 6 is divided into 3 × 3 total 9 regions. Therefore, the position information corresponding to each feature point may be the area ID where each feature point is located, in which case the position information of table 1 is shown in table 2:
| image ID | Hash value | Global features | Local features | |
| ID | ||||
| 1 | |
|
Local feature 1-1 | |
| Global feature 2 | Local features 1-2 | Region 6 | ||
| Global feature 3 | ... | ... | ||
| ... | Local feature 2-1 | Region 7 | ||
| Global feature M | ... | ... | ||
| Local feature 3-1 | Region 2 | |||
| ... | ... | |||
| Local feature N-1 | Region 3 | |||
| ID 2 | ... | ... | ... | ... |
TABLE 2
The preprocessing portion 100 may also include a matching image acquisition module for acquiring new matching images from other image libraries or from a network and storing the new matching images in the matching image library 105. Each time a new matching image is stored in the matching image library 105, the respective modules of the preprocessing section 100 execute the flow shown in fig. 5, and store the hash value of the new matching image and various kinds of features and position information in the feature library 104 so that the new matching image can be matched with the query image.
FIG. 8 provides a flowchart of an online processing method performed by the online processing section 200.
S401, the query interface/interface 207 obtains a query image input by a user, and sends the query image to the local feature extraction module 201.
S402, the local feature extraction module 201 extracts the local features of the query image and the location information of each local feature, and sends the local features of the query image and the location information of each local feature to the third matching module 206. S402 refers to S302.
S403, the query interface 207 sends the query image to the global feature extraction module 202.
S404/S406, the global feature extraction module 202 extracts global features in the query image and sends the global features to the hash module 203 and the second matching module 205. S404/S406 refers to S304/S306.
S405, the hash module 203 hashes the global feature to generate a hash value, and sends the hash value to the first matching module 204. S405 refers to S305 and the related description of the hash operation.
S407/S408, the first matching module 204 obtains the hash value of the matching image and the ID of the matching image from the feature repository 104. After obtaining the hash value of the matching image and the ID of the matching image, the first matching module 204 matches the hash value of the query image and the hash value of the matching image, and sends the ID of the matching image in the matching as a first candidate set to the second matching module 205 according to the matching result. The way in which the first matching module 204 matches the hash value of the query image and the hash value of the matching image may be to calculate the similarity between the hash value of the query image and the hash value of each matching image. The first matching module 204 takes the ID of the matching image corresponding to the hash value whose similarity to the hash value of the query image is greater than the first threshold value as the first candidate set. Alternatively, the first matching module 204 uses IDs of matching images corresponding to Q hash values having the greatest similarity to the hash value of the query image in the matching image library 105 as the first candidate set, where Q is a positive integer.
The first matching module 204 is used for primarily screening the matched images in the matched image library 105, so that a large number of images with low similarity to the query image are screened out, and the matching speed and the matching precision of the query image are improved.
S409, the second matching module 205 obtains global features corresponding to the matching image IDs from the feature library 104 according to the matching image IDs in the first candidate set.
S410, the second matching module 205 matches the global feature of the query image with the obtained global feature of the matching image, and sends the ID of the matching image in the matching as a second candidate set to the third matching module 205 according to the matching result. The way in which the second matching module 205 matches the global features of the query image and the global features of the matching images in the first candidate set may be to calculate the similarity between the global features of the query image and the global features of the matching images, i.e. the matching result includes the similarity between the global features of the query image and the global features of the matching images. The second matching module 205 takes the ID of the matching image corresponding to the global feature whose similarity of the global feature of the query image is greater than the second threshold as the second candidate set. Or, the second matching module 205 takes the ID of the matching image corresponding to the W groups of global features in the first candidate set, where the similarity between the global features of the query image is the maximum, as the second candidate set, and W is a positive integer.
The second matching module 205 further screens the matching images in the first candidate set, and further screens out images with lower similarity to the query image from the first candidate set, so that the calculation amount of subsequent matching is reduced, and the matching speed and the matching precision of the query image are further improved.
S411, the third matching module 206 obtains the local feature and the location information corresponding to the matching image ID from the feature library 104 according to the matching image ID in the second candidate set.
S412, the third matching module 206 matches the local feature of the query image and the corresponding location information with the obtained local feature of the matching image and the corresponding location information, determines the ID of the matching image according to the matching result, and obtains the corresponding matching image from the matching image library 105 as the retrieval result according to the ID of the matching image.
S412 includes two parts, the first part being a match of local features and the second part being a match of location information of local features on the match. And matching the first part and the second part of the query image and each matching image in the second candidate set to judge which matching images corresponding to the IDs in the second candidate set can be used as the retrieval result.
First, the third matching module 206 calculates the similarity of each local feature of the query image to each local feature of the matching image x in the second candidate set, the ID of the matching image x being any one of the second candidate set. If the local feature with the maximum similarity to the local feature 1 in the query image in the local features of the matching image x is the local feature 1-1, the local feature with the second highest similarity to the local feature 1 in the query image in the local features of the matching image x is the local feature 1-2, and the similarity between the local feature 1 and the local feature 1-1 is d1The similarity between the local feature 1 and the local feature 1-2 is d2If d is1And d2Ratio ofAnd if the local feature 1 is smaller than the third threshold, the local feature 1 is successfully matched with the local feature 1-1. If d is1And d2Is not less than the third threshold, then the local feature 1 of the query image has no matching local feature among the local features of the matching image x. After the first portion is completed, the third matching module 206 obtains the local feature set on the match of the query image and each matching image. Taking the example that the query image and the matching image x are matched with 10 local features, the 10 local features on the query image and the matching image x are marked as v1To v10And the 10 local features matched with the query image x are recorded as V1To V10Wherein v is1To v10And V1To V10And matching in a one-to-one correspondence manner.
Second part according to v respectively1To v10Location information and V in query image1To V10Determining v from the position information in the matching image x1To v10Region distribution and V in query image1To V10The distribution of the regions in the matching image x. If the position information acquired in S402 and S411 is a coordinate, v needs to be determined from the coordinate at this time1To v10Respectively belonging to which region of the query image and determining V from the coordinates1To V10Respectively to which region of the matching image x. According to v1To v10Generating a region distribution vector f according to the region distribution in the query imagei＝[c1,c2,...,cH]Wherein c isiV included in a region i representing a query image1To v10The number of the corresponding characteristic points is that i is more than or equal to 1 and less than or equal to H. According to V1To V10Generating a region distribution vector F in the matching imagei＝[C1,C2,...,CH]In which C isiV included in region i representing a matching image1To V10The number of the corresponding characteristic points is that i is more than or equal to 1 and less than or equal to H. Taking FIG. 9 as an example, fi＝[2，1，1，1，0，1，1，3，0]，Fi＝[2，2，0，1，0，1，1，2，1]. Calculating fiAnd FiIf f is the matching resultiAnd FiIs greater than the fourth threshold, the matching image x may be taken as the retrieval result. The third matching module 206 obtains the corresponding matching image from the matching image library 104 according to I D of the matching image x as the retrieval result.
The retrieval result is further screened out from the second candidate set through the first part and the second part, and the accuracy of the retrieval result is further improved. The adoption of the position information of the local features ensures that the query image and the image in the retrieval result have more local feature matching, and meanwhile, the positions of the feature points corresponding to the matched local features in the image are similar, thereby ensuring the similarity between the query image and the matched image.
Optionally, in S412, the search result may be selected from the second candidate set through the first part and the second part. The retrieval result may also be selected from the second candidate set without using the second part, i.e. without using the location information of the local feature. In this case, the local feature extraction module 101/201, the third matching module 206, and the feature library 104 in fig. 5 and 8 do not need to generate, transmit, and store the location information of the local feature. And under the condition of only adopting the first part, after the number of the local features matched between the query image and the matching image x is obtained, judging whether the number of the matched local features is greater than a fifth threshold value or not, or whether the ratio of the number of the matched local features to the total number of the local features in the query image is greater than a sixth threshold value or not. If the number of local features on the match is greater than the fifth threshold, or the ratio of the number of local features on the match to the total number of local features in the query image is greater than the sixth threshold, the matching image x may be used as the retrieval result.
S413, the third matching module 206 returns the retrieved result to the user through the query interface 207.
In the above steps, S401 to S402, S403 to S406 may be executed in parallel. S404 to S405, S406 may be executed in parallel. S407, S409, and S411 may be executed in parallel, or may be executed by combining the hash value, the matching image ID, the global feature, the local feature, and the location information together in S407, where in S408, the first matching module 204 sends I D, the global feature, the local feature, and the location information of the matching image to the second matching module 205 in addition to sending the first candidate set to the second matching module 205. In S410, the second matching module 205 sends I D, local features and location information of the matching image to the third matching module 206 in addition to sending the second candidate set to the third matching module 206.
The present application further provides a preprocessing apparatus 500 for executing the preprocessing portion 100, as shown in fig. 10, the preprocessing apparatus 500 includes a local feature extraction module 101, a global feature extraction module 102, a hash module 103, a feature library 104 and a matching image library 105. The present application further provides an image retrieving apparatus 600 for executing the online processing portion 200, as shown in fig. 11, the image retrieving apparatus 600 includes a local feature extracting module 201, a global feature extracting module 202, a hashing module 203, a first matching module 204, a second matching module 205, and a third matching module 206. Each module in the preprocessing device 500 and the image retrieval device 600 may be a software module, and the feature library 104 may be one database or a storage space provided by a cloud storage service.
The present application further provides an image retrieval system 700. As shown in fig. 12, the image retrieval system 700 includes a preprocessing cluster for executing the preprocessing portion 100 and an online processing cluster for processing the online processing portion 200, each cluster including at least one computing device 705. The computing device 705 includes a bus 703, a processor 701, a communication interface 702, and a memory 704. The processor 701, the memory 704 and the communication interface 702 communicate with each other via a bus 703.
The processor 701 may be a Central Processing Unit (CPU). The memory may include a volatile memory (RAM), such as a Random Access Memory (RAM). The memory 704 may also include a non-volatile memory (english: non-volatile memory), such as a read-only memory (english: read-only memory, abbreviation:ROM), flash memory, HDD, or SSD. The memory 704 has stored therein executable program code that the processor 701 executes to perform the methods of fig. 5 and 8 described above. The memory 704 may also include other software modules required to run a process, such as an operating system. The operating system may be LINUXTM,UNIXTM,WINDOWSTMAnd the like.
The memory 704 of the computing device 705 stores codes required for running the respective modules of the preprocessing unit 500 and the image retrieval unit 600, and the processor 701 executes the codes to realize the functions of the preprocessing unit 500 and the image retrieval unit 600, that is, to execute the methods shown in fig. 5 and 8. The computing devices 700 in the pre-processing cluster may be computing devices in a cloud environment. The computing devices 700 in the online processing cluster may be computing devices in a cloud environment, or computing devices in an edge environment or computing devices in a user environment. The feature library 104 and the matching image library 105 occupy large storage resources and may also be stored in a storage cluster in the preprocessing cluster. The online processing system communicates with a user's query interface 207, obtains query images input by the user and returns search results to the user.
The descriptions of the flows corresponding to the above-mentioned figures have respective emphasis, and for parts not described in detail in a certain flow, reference may be made to the related descriptions of other flows.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in, or transmitted from one computer-readable storage medium to another computer-readable storage medium, the computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more available media, such as a magnetic medium (e.g., floppy disks, hard disks, magnetic tapes), an optical medium (e.g., DVDs), or a semiconductor medium (e.g., SSDs), etc.
Claims (10)
1. An image retrieval method performed by a computing device, comprising:
acquiring a hash value of the global feature of a query image, matching the hash value of the global feature of the query image with the hash value of the global feature of a matching image library;
selecting a first candidate set from the matching images in the matching image library according to the matching result of the hash value of the global feature of the query image and the hash value of the global feature of the matching images in the matching image library;
acquiring global features of the query image, and matching the global features of the query image with the global features of the matched images in the first candidate set;
selecting a second candidate set from the matching images of the first candidate set according to the global features of the query image and the matching results of the global features of the matching images in the first candidate set;
obtaining local features of the query image, and matching the local features of the query image with the local features of the matched images in the second candidate set;
and determining a retrieval result from the matching images in the second candidate set according to the local features of the query image and the matching results of the local features of the matching images in the second candidate set.
2. The method of claim 1, further comprising: acquiring the regional information of the local features of the query image in the query image;
the local features of the query image are obtained and matched with the local features of the query image and the local features of the matched images in the second candidate set; determining a retrieval result from the matching images of the second candidate set according to the matching result of the local feature of the query image and the local feature of the matching images of the second candidate set comprises:
matching region information of local features of the query image with region information of local features of matching images in the second candidate set;
and determining the retrieval result from the matching images of the second candidate set according to the matching result of the region information of the local feature of the query image and the region information of the local feature of the matching images in the second candidate set, wherein the similarity between the region information of the local feature of the matching images in the retrieval result and the region information of the local feature of the query image is greater than a first threshold value.
3. The method of claim 1, further comprising: acquiring the regional information of the local features of the query image in the query image;
the local features of the query image are obtained and matched with the local features of the query image and the local features of the matched images in the second candidate set; determining a retrieval result from the matching images of the second candidate set according to the matching result of the local feature of the query image and the local feature of the matching images of the second candidate set comprises:
matching local features of the query image with local features of matching images in the second candidate set;
acquiring a first region distribution vector of the local features of the matched query image in the query image according to the region information of the local features of the query image;
acquiring a second region distribution vector of the local features of the matched images of the second candidate set in the matched images of the second candidate set according to the region information of the local features of the matched images of the second candidate set;
and judging whether the similarity of the first region distribution vector and the second region distribution vector is greater than a second threshold value, and if the similarity of the first region distribution vector and the second region distribution vector is greater than the second threshold value, determining that the matched image in the second candidate set is included in the retrieval result.
4. An image retrieval apparatus, comprising:
the first matching module is used for acquiring the hash value of the global feature of the query image, matching the hash value of the global feature of the query image with the hash value of the global feature of the matched image library; selecting a first candidate set from the matching images in the matching image library according to the matching result of the hash value of the global feature of the query image and the hash value of the global feature of the matching images in the matching image library; sending the first candidate set to a second matching module;
the second matching module is used for acquiring the global features of the query image and matching the global features of the query image with the global features of the matched images in the first candidate set; selecting a second candidate set from the matching images of the first candidate set according to the global features of the query image and the matching results of the global features of the matching images in the first candidate set; sending the second candidate set to a third matching module;
the third matching module is configured to obtain the local feature of the query image, and match the local feature of the query image with the local feature of the matched image in the second candidate set; and determining a retrieval result from the matching images in the second candidate set according to the local features of the query image and the matching results of the local features of the matching images in the second candidate set.
5. The apparatus of claim 4, wherein the third matching module is to:
acquiring the regional information of the local features of the query image in the query image; matching region information of local features of the query image with region information of local features of matching images in the second candidate set; and determining the retrieval result from the matching images of the second candidate set according to the matching result of the region information of the local feature of the query image and the region information of the local feature of the matching images in the second candidate set, wherein the similarity between the region information of the local feature of the matching images in the retrieval result and the region information of the local feature of the query image is greater than a first threshold value.
6. The apparatus of claim 4, wherein the third matching module is to:
acquiring the regional information of the local features of the query image in the query image; matching local features of the query image with local features of matching images in the second candidate set; acquiring a first region distribution vector of the local features of the matched query image in the query image according to the region information of the local features of the query image; acquiring a second region distribution vector of the local features of the matched images of the second candidate set in the matched images of the second candidate set according to the region information of the local features of the matched images of the second candidate set; and judging whether the similarity of the first region distribution vector and the second region distribution vector is greater than a second threshold value, and if the similarity of the first region distribution vector and the second region distribution vector is greater than the second threshold value, determining that the matched image in the second candidate set is included in the retrieval result.
7. A computing device system comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device being configured to execute program code in the memory to perform the method of any of claims 1 to 3.
8. A non-transitory readable storage medium, wherein the non-transitory readable storage medium, when executed by a computing device, performs the method of any of claims 1 to 3.
9. An image matching method performed by a computing device, comprising:
acquiring the regional information of at least one local feature of a query image and the regional information of at least one local feature of a matching image;
acquiring a first region distribution vector of at least one local feature of the query image in the query image according to region information of the at least one local feature of the query image, wherein the first region distribution vector comprises the number of local features included in each region of the query image;
acquiring a second region distribution vector of at least one local feature of the matched image in the matched image according to region information of the at least one local feature of the matched image, wherein the second region distribution vector comprises the number of local features included in each region of the matched image;
determining that a similarity of the first region distribution vector and the second region distribution vector is greater than a threshold;
determining that the query image matches the matching image.
10. A computing device system comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device to execute program code in the memory to perform the method of claim 9.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811568022.2A CN111353062A (en) | 2018-12-21 | 2018-12-21 | Image retrieval method, device and equipment |
| PCT/CN2019/106737 WO2020125100A1 (en) | 2018-12-21 | 2019-09-19 | Image search method, apparatus, and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811568022.2A CN111353062A (en) | 2018-12-21 | 2018-12-21 | Image retrieval method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111353062A true CN111353062A (en) | 2020-06-30 |
Family
ID=71102024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811568022.2A Pending CN111353062A (en) | 2018-12-21 | 2018-12-21 | Image retrieval method, device and equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111353062A (en) |
| WO (1) | WO2020125100A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113160167A (en) * | 2021-04-16 | 2021-07-23 | 重庆飞唐网景科技有限公司 | Medical image data extraction working method through deep learning network model |
| CN114359590A (en) * | 2021-12-06 | 2022-04-15 | 支付宝(杭州)信息技术有限公司 | NFT image work infringement detection method and device and computer storage medium |
| WO2022111070A1 (en) * | 2020-11-26 | 2022-06-02 | Oppo广东移动通信有限公司 | Image processing method and apparatus, and electronic device and storage medium |
| WO2023272659A1 (en) * | 2021-06-30 | 2023-01-05 | 东莞市小精灵教育软件有限公司 | Method and apparatus for recognizing cover image, storage medium, and recognition device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542058A (en) * | 2011-12-29 | 2012-07-04 | 天津大学 | Hierarchical landmark identification method integrating global visual characteristics and local visual characteristics |
| CN103995864A (en) * | 2014-05-19 | 2014-08-20 | 深圳先进技术研究院 | Image retrieval method and device |
| CN105069089A (en) * | 2015-08-04 | 2015-11-18 | 小米科技有限责任公司 | Picture detection method and device |
| CN107239535A (en) * | 2017-05-31 | 2017-10-10 | 北京小米移动软件有限公司 | Similar pictures search method and device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8280196B2 (en) * | 2009-05-12 | 2012-10-02 | Canon Kabushiki Kaisha | Image retrieval apparatus, control method for the same, and storage medium |
| CN102368237B (en) * | 2010-10-18 | 2013-03-27 | 中国科学技术大学 | Image retrieval method, device and system |
| CN103177105B (en) * | 2013-03-26 | 2016-12-28 | 新浪网技术(中国)有限公司 | A kind of image search method and device |
| CN105608230B (en) * | 2016-02-03 | 2019-05-31 | 南京云创大数据科技股份有限公司 | A kind of Business Information recommender system and method based on image retrieval |
| CN106383891B (en) * | 2016-09-22 | 2019-04-23 | 重庆理工大学 | A kind of medical image distributed search method based on depth Hash |
-
2018
- 2018-12-21 CN CN201811568022.2A patent/CN111353062A/en active Pending
-
2019
- 2019-09-19 WO PCT/CN2019/106737 patent/WO2020125100A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542058A (en) * | 2011-12-29 | 2012-07-04 | 天津大学 | Hierarchical landmark identification method integrating global visual characteristics and local visual characteristics |
| CN103995864A (en) * | 2014-05-19 | 2014-08-20 | 深圳先进技术研究院 | Image retrieval method and device |
| CN105069089A (en) * | 2015-08-04 | 2015-11-18 | 小米科技有限责任公司 | Picture detection method and device |
| CN107239535A (en) * | 2017-05-31 | 2017-10-10 | 北京小米移动软件有限公司 | Similar pictures search method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022111070A1 (en) * | 2020-11-26 | 2022-06-02 | Oppo广东移动通信有限公司 | Image processing method and apparatus, and electronic device and storage medium |
| CN113160167A (en) * | 2021-04-16 | 2021-07-23 | 重庆飞唐网景科技有限公司 | Medical image data extraction working method through deep learning network model |
| CN113160167B (en) * | 2021-04-16 | 2022-01-14 | 深圳市铱硙医疗科技有限公司 | Medical image data extraction working method through deep learning network model |
| WO2023272659A1 (en) * | 2021-06-30 | 2023-01-05 | 东莞市小精灵教育软件有限公司 | Method and apparatus for recognizing cover image, storage medium, and recognition device |
| CN114359590A (en) * | 2021-12-06 | 2022-04-15 | 支付宝(杭州)信息技术有限公司 | NFT image work infringement detection method and device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020125100A1 (en) | 2020-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111353062A (en) | Image retrieval method, device and equipment | |
| CN109710780B (en) | Archiving method and device | |
| Ibrahim et al. | Cluster representation of the structural description of images for effective classification | |
| EP2805262B1 (en) | Image index generation based on similarities of image features | |
| CN108881947B (en) | Method and device for detecting infringement of live stream | |
| CN106055573B (en) | Shoe print image retrieval method and system under multi-instance learning framework | |
| CN107330027B (en) | Weak supervision depth station caption detection method | |
| CN111651636A (en) | Video similar segment searching method and device | |
| US20130046793A1 (en) | Fast matching of image features using multi-dimensional tree data structures | |
| Zhou et al. | Region convolutional features for multi-label remote sensing image retrieval | |
| Aslam et al. | Image classification based on mid-level feature fusion | |
| CN115082790A (en) | Remote sensing image scene classification method based on continuous learning | |
| Bai et al. | Principal pixel analysis and SVM for automatic image segmentation | |
| CN111209940A (en) | Image duplicate removal method and device based on feature point matching | |
| CN113536020B (en) | Method, storage medium and computer program product for data query | |
| CN114860991A (en) | Short video de-duplication method and computer readable storage medium | |
| CN113360911A (en) | Malicious code homologous analysis method and device, computer equipment and storage medium | |
| US8971669B2 (en) | Method and apparatus for image processing | |
| CN112508000A (en) | Method and equipment for generating OCR image recognition model training data | |
| CN108830302B (en) | Image classification method, training method, classification prediction method and related device | |
| CN113657378B (en) | Vehicle tracking method, vehicle tracking system and computing device | |
| Chang et al. | A square-root sampling approach to fast histogram-based search | |
| CN112906824B (en) | Vehicle clustering method, system, device and storage medium | |
| US20230260262A1 (en) | Automated annotation of visual data through computer vision template matching | |
| CN114299012A (en) | Object surface defect detection method and system based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220225 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| TA01 | Transfer of patent application right |