WO2023273082A1

WO2023273082A1 - Method for constructing feature base library, feature retrieval method, and related device

Info

Publication number: WO2023273082A1
Application number: PCT/CN2021/128681
Authority: WO
Inventors: Kaihang YANG; Shulei ZHU; Xingming Zhang
Original assignee: Zhejiang Dahua Technology Co., Ltd.
Priority date: 2021-07-02
Filing date: 2021-11-04
Publication date: 2023-01-05
Also published as: CN113255610B; CN113255610A

Abstract

A method for constructing a feature base library, a feature retrieval method, and related device are provided. The method for constructing a feature base library includes: a first entry feature of a highest target layer is determined, from a layer above the highest target layer of a to-be-inserted feature in the feature base library; a first matching feature of the to-be-inserted feature is searched for in the highest target layer, based on the first entry feature; a second entry feature of the layer above the highest target layer is determined, in response to the number of the first matching feature in the layer above the highest target layer not reaching a preset number, and a second matching feature of the to-be-inserted feature is searched for in the layer above the highest target layer, based on the second entry feature. A recall rate of the feature retrieval is improved.

Description

METHOD FOR CONSTRUCTING FEATURE BASE LIBRARY, FEATURE RETRIEVAL METHOD, AND RELATED DEVICE

TECHNICAL FIELD

The present disclosure relates to the technical field of image retrieval, and in particular to a method for constructing a feature base library, a feature retrieval method, and related device.

BACKGROUND

A feature retrieval technology refers to performing a similarity alignment based on feature vectors acquired via a feature extraction technique to find similar features of a to-be-retrieved feature, and then finding corresponding pictures based on the similar features. In general, when base library data is not a lot, a feature retrieval may be performed in a linear brute force search manner. However, with the development of science and technology, especially in the field of security and monitoring, hundreds of billions of data have been accumulated, and feature dimensions of vectorization are as high as hundreds of thousands. Linear search can no longer meet current needs due to resource occupation and efficiency issues. Therefore, in terms of feature retrieval, corresponding indexes and search mechanisms are constantly generated to reduce search space of vectors, and thereby to improve the retrieval speed, in response to a current situation of features with high dimensions and a large data volume.

Taking face features as an example, the current retrieval of face features is mainly aimed at person-ID search in the security field. That is, according to the face ID photos registered by the Public Security Bureau or one-person-one-file in the face cluster business as a base library, and with face images captured by different models of NVRs (Network Video Recorders) or other electronic devices, as to-be-searched face features, the search efficiency of face features may be improved through a reasonable index construction performed on the ID photos base library and a design of a later search manner. Face feature retrieval methods used commonly mainly include trees, vector quantization, hashing, nearest neighbor graphs, and the like. Among them, the nearest neighbor graphs method performs well in a large-scale high-dimensional face feature search. HNSW (Hierarchical Navigable Small World Graph) is a typical representative of the nearest neighbor graphs method, which is obviously superior to other algorithms in terms of both a recall rate and time consumption. It constructs a multi-layer index mainly through an idea of multiple small-world networks, with the intention of retrieving a certain number of similar features of the to-be-retrieved feature through neighboring edges of the networks. Multiple networks, similar to a multi-layer structure of a binary tree, have the retrieval areas reduced in turn from top to bottom, thereby reducing the retrieval amount and improving retrieval efficiency.

However, the HNSW may involve connectivity and instability of the index structure in all the connections visible, which may result in a low retrieval recall rate and great time consumption of the retrieval.

SUMMARY OF THE DISCLOSURE

The present disclosure provides a method for constructing a feature base library, a feature retrieval method, and related device, which may improve the recall rate of the feature retrieval.

According to a first aspect, a method for constructing a feature base library is provided and includes: determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer; searching, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature; and determining a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and searching, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer.

In some embodiments, the determining a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and searching, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer, further includes: determining the second entry feature of the layer above the highest target layer, in response to the layer above the highest target layer not having the to-be-inserted feature and the number of the first matching feature in the layer above the highest target layer not reaching the preset number; and searching, based on the second entry feature, for the second matching feature of the to-be-inserted feature in the layer above the highest target layer.

The method for constructing a feature base library further includes: determining a third entry feature of a next layer of the highest target layer, in response to the number of the first matching feature in the layer above the highest target layer reaching the preset number, or in response to the layer above the highest target layer not having the to-be-inserted feature and the number of the first matching feature in the layer above the highest target layer reaching the preset number; and searching, based on the third entry feature, for a third matching feature of the to-be-inserted feature in the next layer of the highest target layer.

In some embodiments, the determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer, includes: determining the first entry feature by performing a search from a highest layer of the feature base library to the layer above the highest target layer in a greedy search manner. In some embodiments, the searching, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature, includes: searching for the first matching feature of the to-be-inserted feature in features associated with the first entry feature in the highest target layer, with the first entry feature as an entry.

In some embodiments, the method for constructing a feature base library further includes: establishing an association between the first matching feature and the to-be-inserted feature.

In some embodiments, the determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer, includes: obtaining several features by performing a feature extraction on images in a database; normalizing extracted features; and constructing the feature base library based on normalized features.

According to a second aspect, a feature retrieval method is provided and includes: acquiring a to-be-retrieved feature; determining, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, the first feature set including a plurality of features matching the to-be-retrieved feature; the feature base library being constructed by the above mentioned method for constructing a feature base library; and taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.

In some embodiments, before the determining, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, includes: retrieving, from a highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the highest layer of the feature base library from a preset search entry, and taking the feature with a highest matching degree to the to-be-retrieved feature as a current search entry for a next layer of the highest layer of the feature base library; and retrieving, from the next layer of the highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the next layer of the highest layer of the feature base library from the current search entry.

In some embodiments, the taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library, includes: calculating similarities between each of the features in the first feature set and the to-be-retrieved feature; and selecting at least two of the features with a highest similarity from the first feature set as the search entry of the lowest layer of the feature base library, and retrieving the second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.

In some embodiments, the taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library, includes: taking at least two of the features in the first feature set as the search entry of the lowest layer of the feature base library, while retrieving the second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.

According to a third aspect, a device for constructing a feature base library is provided and includes: an entry determination module, configured to determine, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer; a first search module, configured to search, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature; and a second search module, configured to determine a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and search, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer.

According to a fourth aspect, a feature retrieval device is provided and includes: an acquisition module, configured to acquire a to-be-retrieved feature; a first retrieval module, configured to determine, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, the first feature set including a plurality of features matching the to-be-retrieved feature; the feature base library being constructed by the above mentioned method for constructing a feature base library; and a second retrieval module, configured to take at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.

According to a fifth aspect, an electronic device is provided and includes: a memory, configured to store program instructions; and a processor, configured to call the program instructions stored in the memory to execute the above mentioned method for constructing a feature base library and/or the above mentioned feature retrieval method.

According to a sixth aspect, a computer-readable non-transitory storage medium is provided and configured to store program files, when being executed, the program files are configured to implement the above mentioned method for constructing a feature base library and/or the above mentioned feature retrieval method.

Advantageous effects according to the present disclosure, different from the circumstances of the related art, lie in that: searching, in a highest target layer, for a first matching feature of a to-be-inserted feature, based on a first entry feature of the highest target layer; in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number, searching, based on a second entry feature of the layer above the highest target layer, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer; such that the recall rate of the feature retrieval may be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic flow chart of a method for constructing a feature base library according to an embodiment of the present disclosure.

FIG. 2 is a structural schematic view of a device for constructing a feature base library according to a first embodiment of the present disclosure.

FIG. 3 is a schematic flow chart of a feature retrieval method according to an embodiment of the present disclosure.

FIG. 4 is a structural schematic view of a feature retrieval device according to a second embodiment of the present disclosure.

FIG. 5 is a structural schematic view of an electronic device according to an embodiment of the present disclosure.

FIG. 6 is a structural schematic view of a computer-readable non-transitory storage medium according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The related art provides an approximate nearest neighbor search method based on a hierarchical navigable small world graph. A key design of this method is to modify the insertion manner and lamination manner of new features on the basis of the hierarchical navigable small world graph (HNSW) , and to improve the search speed and stability by solving the imbalance problem of the HNSW index structure. Specifically, in the HNSW index construction process, the method is configured to check whether any of neighbor features of a current inserted feature in a current layer exists in a layer above the current layer, to confirm whether the current feature is promoted to the above layer. According to the innovation, the imbalance of the index structure constructed by the HNSW algorithm and the instability of a search speed may be effectively compensated; whereas simply a HNSW index construction phase taken into account, with a subsequent search phase neglected. Further, there is still improvement room for the method and also a possibility that its search recall rate and performance are further enhanced.

The related art further provides a processing method and device for high-dimensional features. A key design of the method is to combine a clustering algorithm and the HNSW algorithm to reduce a data scale during the HNSW insertion and retrieval process, thereby reducing time consumption of the index construction and retrieval. Firstly, a clustering manner is used to coarsely quantify sample features to generate multiple cluster centers, and then to-be-inserted features are added to corresponding HNSW clusters respectively according to a principle of a closest distance. In the search phase, which is a similar case of the insertion phase, the HNSW algorithm retrieval is performed in a preset number of clusters whose distance is the closest in the sorting to improve the retrieval efficiency. However, in the actual face high-dimensional feature scenario, clustering algorithms such as K-Means have natural defects, which may cause the number of features in the clusters uneven. Further, the first K matching features in the HNSW clusters are not subordinate to the first K matching features of the to-be-searched feature, which affects the recall rate of the first K matching features of the to-be-searched feature.

The related art further provides a data retrieval method based on a regional hierarchical route map algorithm. A key design of the method is to voluntarily select the area size for each level in the map and the number of levels in the entire map based on invariance of the HNSW framework, to initialize the HNSW connection graph; and the algorithm stability and retrieval efficiency are improved by solving the randomness problem of layer number selection in the composition phase of the HNSW algorithm. The method is mainly to perform a voluntarily selection for the level number of the route map, the area size of each level, and the data distribution. Specifically, the layer number of the index structure and the data size of each layer are determined in advance; the layer numbers of all the data of the base library are determined by a clustering manner; the index structure is established based on a manner similar to the HNSW node insertion and subsequently a search for the first K matching features is performed, such that the uncertainty problem of determining the layer number of nodes in the composition stage of HNSW relying on random numbers is solved; and the algorithm stability is improved. However, according to this method, only a stratifying process for the existing data blocks of the base library is performed at one time and the node number of each layer is determined; how to determine the layer number of nodes when the base library data is iteratively increased is not indicated, and whether the maximum layer number should also be updated accordingly when the base library data increases to a certain scale. In addition, it fails to point out any optimization suggestions during the search phase.

In summary, the related art all remains to be improved. The present disclosure provides a feature retrieval method, which makes improvements in both the construction phase and feature retrieval phase of HNSW, so as to enhance the recall rate of the feature retrieval and reduce time consumption of the feature retrieval. The technical solutions in the embodiments of the present disclosure will be clearly and comprehensively described in the following by referring to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of, but not all of, the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without making creative work shall fall within the scope of the present disclosure.

Reference is made to FIG. 1, which is a schematic flow chart of a method for constructing a feature base library according to an embodiment of the present disclosure.

In an operation S11, a first entry feature of a highest target layer is determined from a layer above the highest target layer of a to-be-inserted feature in the feature base library.

Specifically, the feature base library is constructed based on features in a database. Therefore, it is necessary that a feature extraction is performed on images in the database, such as face images, to obtain several features for constructing the feature base library. In a specific embodiment, a deep convolutional network algorithm is used to perform the feature extraction on the images in the database to obtain high-dimensional floating-point type data features (generally 256 dimensions or 512 dimensions) . After the features are obtained, the features and the images corresponding to the features are stored according to their correspondence relationships, so that in the retrieval phase, the corresponding images may be found according to the features.

In a specific embodiment, in order to facilitate the similarity calculation, the extracted features are further normalized, and the normalized features are used to construct the feature base library.

Specifically, before the feature base library is constructed, an int-type parameter M and an int-type parameter efconstruction are needed to be set first, according to the normalized features. The parameter M represents the number of friend points of each node, the larger the better, generally between 5 and 48; the magnitude of M may affect the memory consumption and a recall rate. The parameter efconstruction represents the size of the dynamic candidate node set, and the magnitude of efconstruction affects the index construction speed and index effect. The parameter efconstruction is required to be greater than or equal to the number of friend points M. Under normal circumstances, the effect is better when M is usually set to 50 and efconstriction is set to 500 in the case of 256-dimensional features. There are still some other parameters that are not listed one by one, which are all performed a self-adaption through M. Currently, the index structure involves that the number of connections in the layers except for layer 0 and the maximum number of connections are 50, while the number of connections of layer 0 is 2M, i.e., 100.

A feature is selected, from the features, as a to-be-inserted feature, and a highest target layer C of the to-be-inserted feature is determined by using exponential decay function as follows:

1=-ln (unif (0, 1) ) ·ML, where mL=1/logM.

Specifically, the data amount of each layer will decrease layer by layer from the lowest layer to the highest layer, and in a form of M exponent, which means that a retrieval intensity increases level by level from a loose level to a tight level when a feature retrieval is performed from the higher layer to the lower layer.

After a highest target layer C of the to-be-inserted feature in the feature base library is determined, a first entry feature Q of the highest target layer C is determined in a layer above the highest target layer C, i.e., a C+1 layer, in the feature base library. Specifically, a search from the highest level L of the feature base library to the layer above the highest target layer C, i.e., the C+1 layer, is performed in a greedy search manner, so as to obtain the first entry feature. Specifically, when the layer number of the highest layer L of the feature base library is 5, and the layer number of the highest target layer C is 2, a feature with the highest similarity to the to-be-inserted feature is searched for in the fifth layer, to serve as an entry feature of the fourth layer. With the entry feature of the fourth layer as an entry, a feature with the highest similarity to the to-be-inserted feature is searched for in the features associated with the entry feature of the fourth layer in the fourth layer, to serve as an entry feature of the third layer. With the entry feature of the third layer as an entry, a feature with the highest similarity to the to-be-inserted feature is searched for in the features associated with the entry feature of the third layer in the third layer, to serve as an entry feature of the second layer, i.e., the first entry feature Q of the highest target layer C.

In an operation S12, a first matching feature of the to-be-inserted feature is searched for in the highest target layer, based on the first entry feature.

Specifically, with the first entry feature Q as an entry, a first matching feature of the to-be-inserted feature is searched for in features associated with the first entry feature Q in the highest target layer C. Specifically, when the first matching feature of the to-be-inserted feature is obtained, an association between the first matching feature and the to-be-inserted feature is established. In this way, during the retrieval, the first matching feature associated with the first entry feature Q may be recalled, based on the first entry feature Q.

In an operation S13, a second entry feature of a layer above the highest target layer is determined in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; a second matching feature of the to-be-inserted feature in the layer above the highest target layer is searched for, based on the second entry feature.

In the present disclosure, after the association between the to-be-inserted feature and the first matching feature is established in the highest target layer C, a judgement of whether the number of the first matching feature in the layer above the highest target layer C, i.e., the C+1 layer, has reached a preset number, is further made.

Specifically, in response to the number of the first matching feature in the layer above the highest target layer not reaching the preset number, a second entry feature R of the layer C+1 above the highest target layer is determined, and a second matching feature of the to-be-inserted feature is searched for in the layer C+1 above the highest target layer, based on the second entry feature R.

Specifically, when the layer number of the highest layer L of the feature base library is 5, and the layer number of the highest target layer C is 2, a feature with the highest similarity to the to-be-inserted feature is searched for in the fifth layer, to serve as an entry feature of the fourth layer. With the entry feature of the fourth layer as an entry, a feature with the highest similarity to the to-be-inserted feature is searched for in the features associated with the entry feature of the fourth layer in the fourth layer, to serve as an entry feature of the third layer, i.e., the second entry feature R of the layer C+1 above the highest target layer C.

With the second entry feature R as the entry, a second matching feature of the to-be-inserted feature is searched for in the features associated with the second entry feature R in the layer C+1 above the highest target layer C. Specifically, while obtaining the second matching feature of the to-be-inserted feature, an association between the second matching feature and the to-be-inserted feature is established. In this way, during the retrieval, the matching features associated with the second entry feature R may be recalled, based on the second entry feature R. It could be understood that, the second matching feature of the to-be-inserted feature are features with similarities within a certain range to the to-be-inserted feature.

Further, in an embodiment, whether the layer C+1 above the highest target layer C has a to-be-inserted feature is determined. When the layer C+1 above the highest target layer C does not have the to-be-inserted feature, whether the number of the first matching feature in the layer C+1 above the highest target layer C reaches the preset number, is determined. In response to the layer C+1 above the highest target layer C not having the to-be-inserted feature, and the number of the first matching feature in the layer C+1 above the highest target layer C not reaching the preset number, a second entry feature Q of the layer C+1 above the highest target layer C is determined, and a second matching feature of the to-be-inserted feature is searched for in the layer C+1 above the highest target layer C, based on the second entry feature Q.

In an embodiment, a third entry feature S of a layer C-1 below the highest target layer is determined in response to the number of the first matching feature in the layer above the highest target layer reaching the preset number. A third matching features of the to-be-inserted feature is searched for in the layer C-1 below the highest target layer C, based on the third entry feature S. Specifically, a feature with the highest similarity to the to-be-inserted feature is determined in the highest target layer C, to serve as the third entry feature S of the layer C-1 below the highest target layer C. Based on the third entry feature S, the third matching feature of the to-be-inserted feature is searched for in the layer C-1 below the highest target layer C, and an association between the to-be-inserted feature and the third matching feature is established in the search process.

In an embodiment, in response to the layer C+1 above the highest target layer C not having the to-be-inserted feature, and the number of the first matching feature in the layer C+1 above the highest target layer C reaching the preset number, a third entry feature S of the layer C-1 below the highest target layer C is determined, and a third matching feature of the to-be-inserted feature is searched for in the layer C-1 below the highest target layer C, based on the third entry feature S.

It could be understood that, in an embodiment of the present disclosure, when the layer C+1 above the highest target layer C has the to-be-inserted feature, a third entry feature S of the layer C-1 below the highest target layer C is determined, and a third matching feature of the to-be-inserted feature is searched for in the layer C-1 below the highest target layer C, based on the third entry feature S.

Traversing each of features in the database, which is inserted into the feature base library in the above-mentioned manner, thereby the construction of the feature base library may be completed.

It should be noted that the above-mentioned "preset number" value affects the sparseness degree of the local layers, thereby affecting the overall time consumption and the recall, and required to be weighed according to actual situations.

According to the method of the present embodiment, features and an inclusive relationship between the matching feature of the current layer and features of a layer above the current layer, are taken into account; issues of the retrieval connectivity and stability are further improved, and the recall rate of the feature retrieval is enhanced and the time consumption of the feature retrieval is reduced.

Reference is made to FIG. 2, which is a structural schematic view of a device for constructing a feature base library according to a first embodiment of the present disclosure. The device specifically includes: an entry determination module 21, a first search module 22, and a second search module 23.

The entry determination module 21 is configured to select a feature, from the features, as a to-be-inserted feature, and a highest target layer C of the to-be-inserted feature is determined by using exponential decay function as follows:

1=-ln (unif (0, 1) ) ·ML, where mL=1/logM.

After the entry determination module 21 is configured to determine a highest target layer C of the to-be-inserted feature in the feature base library, the entry determination module 21 is configured to determine a first entry feature Q of the highest target layer C in a layer above the highest target layer C, i.e., a C+1 layer, in the feature base library. Specifically, a search from the highest level L of the feature base library to the layer above the highest target layer C, i.e., the C+1 layer, is performed in a greedy search manner, so as to obtain the first entry feature. Specifically, when the layer number of the highest layer L of the feature base library is 5, and the layer number of the highest target layer C is 2, a feature with the highest similarity to the to-be-inserted feature is searched for in the fifth layer, to serve as an entry feature of the fourth layer. With the entry feature of the fourth layer as an entry, a feature with the highest similarity to the to-be-inserted feature is searched for in the features associated with the entry feature of the fourth layer in the fourth layer, to serve as an entry feature of the third layer. With the entry feature of the third layer as an entry, a feature with the highest similarity to the to-be-inserted feature is searched for in the features associated with the entry feature of the third layer in the third layer, to serve as an entry feature of the second layer, i.e., the first entry feature Q of the highest target layer C.

The first search module 22 is configured to search for a first matching feature of the to-be-inserted feature in the highest target layer, based on the first entry feature.

Specifically, with the first entry feature Q as an entry, the first search module 22 is configured to search for a first matching feature of the to-be-inserted feature in features associated with the first entry feature Q in the highest target layer C. Specifically, when the first matching feature of the to-be-inserted feature is obtained, an association between the first matching feature and the to-be-inserted feature is established. In this way, during the retrieval, the first matching feature associated with the first entry feature Q may be recalled, based on the first entry feature Q.

The second search module 23 is configured to determine a second entry feature of a layer above the highest target layer, in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and search for a second matching feature of the to-be-inserted feature in the layer above the highest target layer, based on the second entry feature.

In the present disclosure, after the association between the to-be-inserted feature and the first matching feature is established in the highest target layer C, the second search module 23 is configured to further judge whether the number of the first matching feature in the layer above the highest target layer C, i.e., the C+1 layer, has reached a preset number.

Specifically, in response to the number of the first matching feature in the layer above the highest target layer not reaching the preset number, the second search module 23 is configured to determine a second entry feature R of the layer C+1 above the highest target layer, and search for a second matching feature of the to-be-inserted feature in the layer C+1 above the highest target layer, based on the second entry feature R.

With the second entry feature R as the entry, the second search module 23 is configured to search for a second matching feature of the to-be-inserted feature, in the features associated with the second entry feature R in the layer C+1 above the highest target layer C. Specifically, while obtaining the second matching feature of the to-be-inserted feature, an association between the second matching feature and the to-be-inserted feature is established. In this way, during the retrieval, the matching features associated with the second entry feature R may be recalled, based on the second entry feature R. It could be understood that, the second matching feature of the to-be-inserted feature are features with similarities within a certain range to the to-be-inserted feature.

Further, in an embodiment, the second search module 23 is configured to determine whether the layer C+1 above the highest target layer C has a to-be-inserted feature. When the layer C+1 above the highest target layer C does not have the to-be-inserted feature, the second search module 23 is configured to further determine whether the number of the first matching feature in the layer C+1 above the highest target layer C reaches the preset number. In response to the layer C+1 above the highest target layer C not having the to-be-inserted feature, and the number of the first matching feature in the layer C+1 above the highest target layer C not reaching the preset number, the second search module 23 is configured to determine a second entry feature Q of the layer C+1 above the highest target layer C, and search for a second matching feature of the to-be-inserted feature in the layer C+1 above the highest target layer C, based on the second entry feature Q.

In an embodiment, the second search module 23 is configured to determine a third entry feature S of a layer C-1 below the highest target layer, in response to the number of the first matching feature in the layer above the highest target layer reaching the preset number; and search for a third matching features of the to-be-inserted feature in the layer C-1 below the highest target layer C, based on the third entry feature S. Specifically, a feature with the highest similarity to the to-be-inserted feature is determined in the highest target layer C, to serve as the third entry feature S of the layer C-1 below the highest target layer C. Based on the third entry feature S, the third matching feature of the to-be-inserted feature is searched for in the layer C-1 below the highest target layer C, and an association between the to-be-inserted feature and the third matching feature is established in the search process.

In an embodiment, in response to the layer C+1 above the highest target layer C not having the to-be-inserted feature, and the number of the first matching feature in the layer C+1 above the highest target layer C reaching the preset number, the second search module 23 is configured to determine a third entry feature S of the layer C-1 below the highest target layer C, and search for a third matching feature of the to-be-inserted feature in the layer C-1 below the highest target layer C, based on the third entry feature S.

It could be understood that, in an embodiment of the present disclosure, when the layer C+1 above the highest target layer C has the to-be-inserted feature, the second search module 23 is configured to determine a third entry feature S of the layer C-1 below the highest target layer C, and search for a third matching feature of the to-be-inserted feature in the layer C-1 below the highest target layer C, based on the third entry feature S.

According to the device of the present embodiment, features and an inclusive relationship between the matching feature of the current layer and features of a layer above the current layer, are taken into account; issues of the retrieval connectivity and stability are further improved, and the recall rate of the feature retrieval is enhanced and the time consumption of the feature retrieval is reduced.

Reference is made to FIG. 3, which is a schematic flow chart of a feature retrieval method according to an embodiment of the present disclosure.

In an operation S31, a to-be-retrieved feature is acquired.

Specifically, features of a to-be-retrieved image are extracted, such that a to-be-retrieved feature is acquired. In an embodiment, in order to facilitate the similarity calculation, the to-be-retrieved feature is further normalized. A retrieving process is performed from the feature base library based on the normalized to-be-retrieved feature, to obtain matching features of the to-be-retrieved feature. The feature base library of the present embodiment is a feature base library constructed through the embodiment shown above in FIG. 1.

In an operation S32, a first feature set matching the to-be-retrieved feature is determined from a layer above a lowest layer of a feature base library, the first feature set includes a plurality of features matching the to-be-retrieved feature.

Specifically, a first feature set matching the to-be-retrieved feature is determined from the layer above the lowest layer of the feature base library. The first feature set includes a plurality of features matching the to-be-retrieved feature.

Specifically, a feature with a highest matching degree to the to-be-retrieved feature is retrieved from a highest layer of the feature base library, by entering the highest layer of the feature base library from a preset search entry. The feature with a highest matching degree to the to-be-retrieved feature is taken as a current search entry for a next layer of the highest layer of the feature base library. A feature with a highest matching degree to the to-be-retrieved feature is retrieved from the next layer of the highest layer of the feature base library, by entering the next layer of the highest layer of the feature base library from the current search entry. In an embodiment, when the highest layer of the feature base library is H, a feature point is randomly selected as a preset search entry for the highest layer H of the feature base library. Based on the preset search entry, a feature with the highest matching degree to the to-be-retrieved feature is retrieved, from the features associated with the preset search entry in the highest layer H of the feature base library, to serve as a search entry of a layer H-1 below the highest layer of the feature base library. Based on the search entry of the layer H-1 below the highest layer of the feature base library, a feature with the highest matching degree to the to-be-retrieved feature is retrieved from the features associated with the search entry of the layer H-1 in the layer H-1 below the highest layer of the feature base library, to serve as a search entry of a layer H-2 of the feature base library. Based on the search entry of the layer H-2 of the feature base library, a feature with the highest matching degree to the to-be-retrieved feature is retrieved, from the features associated with the search entry of the layer H-2 in the layer H-2 below the highest layer of the feature base library, to serve as a search entry of a layer H-3 of the feature base library. The above process is repeated until the layer above the lowest layer of the feature base library is retrieved.

In an embodiment, a first feature set matching the to-be-retrieved feature is determined from the layer above the lowest layer of the feature base library, at the search entry of the layer above the lowest layer of the feature base library.

In an operation S33, at least two of the plurality of features in the first feature set taken as a search entry of a lowest layer of the feature base library, a second feature set matching the to-be-retrieved feature is retrieved from the lowest layer of the feature base library.

Specifically, at least two features of the first feature set are regarded as the search entry, a second feature set matching the to-be-retrieved feature is searched for in the lowest layer of the feature base library.

Specifically, similarities between each of the features in the first feature set and the to-be-retrieved feature are calculated; at least two of the features with a highest similarity from the first feature set are selected as the search entry of the lowest layer of the feature base library; and the second feature set matching the to-be-retrieved feature is retrieved from the lowest layer of the feature base library. For example, there are 20 features in the first feature set matching the to-be-retrieved feature found, among the features associated with the search entry in a layer above the lowest layer of the feature base library. In this case, similarities between the 20 features and the to-be-retrieved feature are calculated, and 10 features with relatively higher similarities are selected as the search entry of the lowest layer of the feature base library.

In a feasible embodiment, two int-type parameters, namely, ef and k, are set, where the parameter ef represents the size of the dynamic candidate feature set during search. Generally, the larger the parameter, the higher the recall rate and the greater the time consumption. The parameter k represents a to-be-searched feature. The dynamic candidate feature set is continuously updated during the search process, and finally k features may be found from the dynamic candidate feature set. Therefore, the number of features in the dynamic candidate feature set ef is required to be greater than or equal to the value of the parameter k.

In an embodiment, when the number of features in the dynamic candidate feature set ef is 100, and the parameter k is 10. With each of features in the first feature set as a search entry, features associated with the each of features are found in the lowest layer of the feature base library; similarities between the found features and the to-be-retrieved feature are calculated. When the number of features in the dynamic candidate feature set ef is less than 100, the found features are added into the dynamic candidate feature set ef with corresponding similarities annotated. When the number of features in the dynamic candidate feature set ef reaches 100, similarities between the found features and the to-be-retrieved feature are calculated; features with relatively less similarities in the dynamic candidate feature set ef are removed, while the features whose similarities are currently calculated are added to the dynamic candidate feature set ef. Through this method, after features associated with each of the features are traversed, 100 features having certain similarities to the to-be-retrieved feature may be obtained. The features whose similarities to the to-be-retrieved feature are ranked top relatively, that is, the first 10 features with the highest similarities, are found from the dynamic candidate feature set ef to serve as a second feature set.

In a specific embodiment, at least two features in the first feature set can be used as a search entry for the lowest layer of the feature base library, while a second feature set matching the to-be-retrieved feature is retrieved from the lowest layer of the feature base library.

In the present embodiment, a search is performed in the lowest layer of the feature base library, with a plurality of features as the search entry, resulting with obtaining the first K features with the highest similarities to the to-be-retrieved feature. Compared with the related-art solution in which one feature is used as a search entry in the lowest layer, the recall rate of features is improved; the final retrieval accuracy of the to-be-retrieved feature is improved to a certain extent. In addition, in the present embodiment, parallel searches are performed at the same time with multiple search entries, interior searches at different search entries are performed independently; and meanwhile, the time consumption may be significantly reduced by the parallel searches.

Reference is made to FIG. 4, which is a structural schematic view of a feature retrieval device according to a second embodiment of the present disclosure. The feature retrieval device includes an acquisition module 41, a first retrieval module 42, and a second retrieval module 43.

The acquisition module 41 is configured to acquire a to-be-retrieved feature. Specifically, the acquisition module 41 is configured to extract features of a to-be-retrieved image, such that a to-be-retrieved feature is acquired. In an embodiment, in order to facilitate the similarity calculation, the to-be-retrieved feature is further normalized, a retrieving process is performed from the feature base library based on the normalized to-be-retrieved feature, to obtain matching features of the to-be-retrieved feature. The feature base library of the present embodiment is a feature base library constructed through the embodiment shown above in FIG. 1.

The first retrieval module 42 is configured to determine a first feature set matching the to-be-retrieved feature from a layer above a lowest layer of a feature base library, the first feature set includes a plurality of features matching the to-be-retrieved feature.

Specifically, the first retrieval module 42 is configured to determine a first feature set matching the to-be-retrieved feature, from the layer above the lowest layer of the feature base library. The first feature set includes a plurality of features matching the to-be-retrieved feature.

Specifically, the first retrieval module 42 is configured to retrieve, from a highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the highest layer of the feature base library from a preset search entry; and take the feature with a highest matching degree to the to-be-retrieved feature as a current search entry for a next layer of the highest layer of the feature base library. The first retrieval module 42 is configured to retrieve, from the next layer of the highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the next layer of the highest layer of the feature base library from the current search entry. In an embodiment, when the highest layer of the feature base library is H, feature points are randomly selected as a preset search entry for the highest layer H of the feature base library; based on the preset search entry, a feature with the highest matching degree to the to-be-retrieved feature is retrieved, from the features associated with the preset search entry in the highest layer H of the feature base library, to serve as a search entry of a layer H-1 below the highest layer of the feature base library. Based on the search entry of the layer H-1 below the highest layer of the feature base library, a feature with the highest matching degree to the to-be-retrieved feature is retrieved, from the features associated with the search entry of the layer H-1 in the layer H-1 below the highest layer of the feature base library, to serve as a search entry of a layer H-2 of the feature base library. Based on the search entry of the layer H-2 of the feature base library, a feature with the highest matching degree to the to-be-retrieved feature is retrieved, from the features associated with the search entry of the layer H-2 in the layer H-2 below the highest layer of the feature base library, to serve as a search entry of a layer H-3 of the feature base library. The above process is repeated until the layer above the lowest layer of the feature base library is retrieved.

In an embodiment, the first retrieval module 42 is configured to determine a first feature set matching the to-be-retrieved feature from the layer above the lowest layer of the feature base library, at the search entry of the layer above the lowest layer of the feature base library.

The second retrieval module 43 is configured to take at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieve a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.

Specifically, the second retrieval module 43 is configured to regard at least two features of the first feature set as the search entry, and search for a second feature set matching the to-be-retrieved feature in the lowest layer of the feature base library.

Specifically, the second retrieval module 43 is configured to calculate similarities between each of the features in the first feature set and the to-be-retrieved feature; and select at least two of the features with a highest similarity from the first feature set as the search entry of the lowest layer of the feature base library; the second retrieval module 43 is configured to retrieve the second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library. For example, there are 20 features in the first feature set matching the to-be-retrieved feature found, among the features associated with the search entry in a layer above the lowest layer of the feature base library. In this case, similarities between the 20 features and the to-be-retrieved feature are calculated, and 10 features with relatively higher similarities are selected as the search entry of the lowest layer of the feature base library.

In a specific embodiment, the second retrieval module 43 is configured to use at least two features in the first feature set as a search entry for the lowest layer of the feature base library, while retrieve, from the lowest layer of the feature base library, the second feature set matching the to-be-retrieved feature.

The retrieval device according to the present embodiment, a search is performed in the lowest layer of the feature base library, with a plurality of features as the search entry, resulting with obtaining the first K features with the highest similarities to the to-be-retrieved feature. Compared with the related-art solution in which one feature is used as a search entry in the lowest layer, the recall rate of features is improved; the final retrieval accuracy of the to-be-retrieved feature is improved to a certain extent. In addition, in the present embodiment, parallel searches are performed at the same time with multiple search entries, interior searches at different search entries are performed independently; and meanwhile, the time consumption may be significantly reduced by the parallel searches.

Reference is made to FIG. 5, which is a structural schematic view of an electronic device according to an embodiment of the present disclosure. The electronic device includes a memory 202 and a processor 201 connected to each other.

The memory 202 is configured to store program instructions for implementing any of device methods above.

The processor 201 is configured to execute the program instructions stored in the memory 202.

The processor 201 may also be referred to as CPU (Central Processing Unit) . The processor 201 may be an integrated circuit chip with signal processing capabilities. The processor 201 may also be a general-purpose processor, a digital signal processor (DSP) , a disclosure specific integrated circuit (ASIC) , a field programmable gate array (FPGA) , or other programmable logic devices, a discrete gate or transistor logic devices, a discrete hardware component. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The memory 202 can be a memory chip, a TF card, or the like, and may store all information in an electronic device of equipment, including original data input, computer programs, intermediate running results, and final running results all stored in the memory. It stores and extracts information according to a location specified by a controller. Thanks to the memory, the electronic device has a memory function and normal operations can be ensured. The memory of the electronic device can be divided into a main memory (internal storage) and an auxiliary memory (external storage) according to its purposes. There are also classification methods that divide the memory into the external memory and the internal memory. The external storage, which is typically a magnetic medium, an optical disc, or the like, can store information for a long time. The internal storage, which refers to a storage component on the mainboard, is configured to store the currently executed data and programs, but is simply configured to temporarily store the programs and data, and the data will be lost when the power supply is turned off or the power is off.

In the several embodiments provided in the present disclosure, it should be understood that, the disclosed methods and devices may be implemented in other manners. For example, the device implementation manners described above are only illustrative. For example, the division of modules or units is only a logical function division, and other division manners are allowed in actual implementation. For example, multiple units or components may be combined or may be integrated into another system, or some features may be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other form.

The units described as separate components may or may not be physically separate, and the components displayed as units may be or may be not physical units, that is, they may be located in one location, or they may also be distributed on multiple network units. Partial or all of the units may be selected according to actual needs to achieve the purpose of the solution of the present implementing manner.

Furthermore, the respective functional units in each of the present embodiments of the present disclosure may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

When the integrated units are implemented in the form of software functional units and sold or used as independent products, the integrated units may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or a part that contributes to the existing technology, or all or part of the technical solution, may be embodied in the form of software products. The computer software products are stored in a storage medium, including several instructions to make a computer device (which may be a personal computer, a system server, a network device, or the like) or a processor execute all or part of operations of the methods in the various embodiments according to the present disclosure.

Reference is made to FIG. 6, which is a structural schematic view of a computer-readable non-transitory storage medium according to an embodiment of the present disclosure. The storage medium according to the present disclosure stores program files 203 capable of implementing all the above methods. The program files 203, which may be stored in the above storage medium in a form of a software product, includes several instructions to enable a computer device (which may be a personal computer, a server, a network device, or the like) or a processor to execute all or part of the operations of the methods in the various embodiments according to the present disclosure. The aforementioned storage devices include: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory) , a random access memory (RAM) , a magnetic disk or an optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

The foregoing are only the embodiments of the present disclosure, and the scope of the present disclosure should not therefore be limited. Any equivalent structure or equivalent process transformation based on contents of the description and drawings of the present disclosure, or directly or indirectly application to other related technical fields, is likewise included within the scope of patent protection of the present disclosure.

Claims

A method for constructing a feature base library, comprising:

determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer;

searching, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature; and

determining a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and searching, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer.
The method according to claim 1, wherein the determining a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and searching, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer, further comprise:

determining the second entry feature of the layer above the highest target layer, in response to the layer above the highest target layer not having the to-be-inserted feature and the number of the first matching feature in the layer above the highest target layer not reaching the preset number; and searching, based on the second entry feature, for the second matching feature of the to-be-inserted feature in the layer above the highest target layer.
The method according to claim 2, further comprising:

determining a third entry feature of a next layer of the highest target layer, in response to the number of the first matching feature in the layer above the highest target layer reaching the preset number, or in response to the layer above the highest target layer not having the to-be-inserted feature and the number of the first matching feature in the layer above the highest target layer reaching the preset number; and searching, based on the third entry feature, for a third matching feature of the to-be-inserted feature in the next layer of the highest target layer.
The method according to claim 1, wherein the determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer, comprises:

determining the first entry feature by performing a search from a highest layer of the feature base library to the layer above the highest target layer in a greedy search manner.
The method according to claim 1, wherein the searching, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature, comprises:

searching for the first matching feature of the to-be-inserted feature in features associated with the first entry feature in the highest target layer, with the first entry feature as an entry.
The method according to claim 5, further comprising:

establishing an association between the first matching feature and the to-be-inserted feature.
The method according to claim 1, wherein the determining, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer, comprises:

obtaining several features by performing a feature extraction on images in a database;

normalizing extracted features; and

constructing the feature base library based on normalized features.
A feature retrieval method, comprising:

acquiring a to-be-retrieved feature;

determining, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, the first feature set comprising a plurality of features matching the to-be-retrieved feature; the feature base library being constructed by the method for constructing a feature base library according to any one of claims 1 to 7; and

taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.
The method according to claim 8, wherein before the determining, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, comprises:

retrieving, from a highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the highest layer of the feature base library from a preset search entry, and taking the feature with a highest matching degree to the to-be-retrieved feature as a current search entry for a next layer of the highest layer of the feature base library; and

retrieving, from the next layer of the highest layer of the feature base library, a feature with a highest matching degree to the to-be-retrieved feature by entering the next layer of the highest layer of the feature base library from the current search entry.
The method according to claim 8, wherein the taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library, comprises:

calculating similarities between each of the features in the first feature set and the to-be-retrieved feature; and

selecting at least two of the features with a highest similarity from the first feature set as the search entry of the lowest layer of the feature base library, and retrieving the second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.
The method according to claim 8, wherein the taking at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library, comprises:

taking at least two of the features in the first feature set as the search entry of the lowest layer of the feature base library, while retrieving the second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.
A device for constructing a feature base library, comprising:

an entry determination module, configured to determine, from a layer above a highest target layer of a to-be-inserted feature in the feature base library, a first entry feature of the highest target layer;

a first search module, configured to search, in the highest target layer, for a first matching feature of the to-be-inserted feature, based on the first entry feature; and

a second search module, configured to determine a second entry feature of the layer above the highest target layer in response to a number of the first matching feature in the layer above the highest target layer not reaching a preset number; and search, based on the second entry feature, for a second matching feature of the to-be-inserted feature in the layer above the highest target layer.
A feature retrieval device, comprising:

an acquisition module, configured to acquire a to-be-retrieved feature;

a first retrieval module, configured to determine, from a layer above a lowest layer of a feature base library, a first feature set matching the to-be-retrieved feature, the first feature set comprising a plurality of features matching the to-be-retrieved feature; the feature base library being constructed by the method for constructing a feature base library according to any one of claims 1 to 7; and

a second retrieval module, configured to take at least two of the plurality of features in the first feature set as a search entry of a lowest layer of the feature base library, and retrieving a second feature set matching the to-be-retrieved feature from the lowest layer of the feature base library.
An electronic device, comprising:

a memory, configured to store program instructions; and

a processor, configured to call the program instructions stored in the memory to execute the method for constructing a feature base library according to any one of claims 1 to 7 and/or the feature retrieval method according to any one of 8 to 11.
A computer-readable non-transitory storage medium, configured to store program files, wherein when being executed, the program files are configured to implement the method for constructing a feature base library according to any one of claims 1 to 4 and/or the feature retrieval method according to any one of claims 8 to 11.