CN107451200B

CN107451200B - Retrieval method using random quantization vocabulary tree and image retrieval method based on same

Info

Publication number: CN107451200B
Application number: CN201710545225.9A
Authority: CN
Inventors: 王晓春
Original assignee: Xian Jiaotong University
Current assignee: Xian Jiaotong University
Priority date: 2017-07-06
Filing date: 2017-07-06
Publication date: 2020-07-28
Anticipated expiration: 2037-07-06
Also published as: CN107451200A

Abstract

The invention discloses a retrieval method using a random quantized vocabulary tree and an image retrieval method based on the same, which comprises the following steps: (1) generating a nearest neighbor search tree, taking all the feature vectors of the whole database as root nodes of a first section, and segmenting downwards; (2) in the second stage, k points are randomly selected from the whole database to serve as cluster centers, then each feature vector is distributed to the cluster center closest to the feature vector according to the selected similarity measurement method, the whole database is divided into k subsets, and downward segmentation is continued; (3) in the third level, for each k clusters obtained from the second level, k feature points are randomly selected from the feature vector pool thereof as the cluster centers of the next level. (4) And (6) repeating. The image retrieval method of the invention overcomes the problem that the establishment of the vocabulary tree in the prior art needs a large amount of time, can establish the vocabulary tree in a short time and meets the real-time requirement.

Description

Retrieval method using random quantization vocabulary tree and image retrieval method based on same

Technical Field

The invention belongs to the technical field of image retrieval, and particularly relates to a retrieval method using a random quantization vocabulary tree and an image retrieval method based on the retrieval method.

Background

In recent years, with the development and popularization of digital technologies, particularly network technologies, and the development of internet of things and computer information acquisition software and hardware technologies, more and more data are acquired and stored, and the speed of quantity acquisition far exceeds the speed of processing the data by the traditional method, and the trend is more and more obvious. Facebook is a world-ranked top photo sharing site, uploading approximately 3.5 million photos each day by 11 months in 2013, while the photo capacity on Facebook alone has reached 250 PB; in the aspect of digital video, the statistics of YouTube in 2013 show that video content is uploaded for more than 72 hours per minute, there are 40 hundred million website video playing requests per day, and these data are still increasing greatly. How to effectively organize, manage and retrieve a large-scale database for such huge data resources and the same massive access requirements becomes a problem which needs to be solved urgently.

In the traditional image retrieval method based on the text, the image is annotated by using keywords, and the image retrieval is changed into the search of the keywords. The obvious disadvantages are: neither computer vision nor artificial intelligence techniques can automatically label text on images, and manual labeling is required. Because of the continuous expansion of the data scale, the speed of manual annotation is far from the expansion speed of image data, and because of the subjectivity and inaccuracy of manual annotation, different people have different understandings on images, so that the annotation on the images does not have a unified standard. To overcome the limitations of text-Based Image Retrieval methods, Content Based-Image Retrieval (CBIR) was emerging in the 90 s of the 20 th century. The method is different from the traditional retrieval means, integrates the image understanding technology, and provides a method for effectively retrieving images from a large-capacity image database according to the requirements of people.

The basic idea of a content-based image retrieval system is to analyze the visual characteristics of an image and to retrieve in connection with the context. The method is realized by adopting an image database to store and manage image data, and then embedding a content-based image retrieval technology as an engine of the database into the image database to provide a content-based image retrieval function. In the existing content-based image retrieval system, low-level image information including contents such as color, texture, shape of an image and spatial relationship between the images is generally used, the similarity between a query image and a target image is calculated, and then retrieval is performed according to the size of the similarity, namely, the matching degree between image features. Therefore, each image in the image library is firstly converted into a point in the image feature space, namely a corresponding feature vector, by adopting feature extraction, and then, the retrieval of the image is carried out according to the feature vector, so that the content-based image retrieval is converted into the retrieval of the feature point in the image feature space.

In the case where the image database is small in size, the most common image feature retrieval method is a sequential scanning method. However, as means for acquiring information are developed and the information demand is increased, the size of the image database is larger and larger, and the conventional sequential scanning method cannot meet the requirement of the user on the retrieval time. Therefore, the data is effectively organized to quickly reduce the retrieval range and improve the retrieval speed, so that an efficient indexing mechanism is established, which is the key point of content-based retrieval.

In related research in the past, researchers have proposed many data indexing methods for specific application fields. However, these data indexing methods are affected by "dimension disaster" in the high-dimensional space when processing high-dimensional data, and their retrieval performance deteriorates to and even worse than sequential scanning performance when the data dimension increases. Since feature vectors extracted from the original image are usually high-dimensional in CBIR studies, indexing of image feature data is inevitably affected by "dimensional disasters". Nister and Stewenius provide a search method based on a vocabulary tree, which has a good search effect in a high-dimensional space, but the method has a long tree building time for the high-dimensional space, and cannot meet the requirement of a modern database on search timeliness. Therefore, establishing an efficient high-dimensional data indexing mechanism aiming at the high-dimensional characteristics of image feature data is an important challenge faced by current image retrieval research.

Disclosure of Invention

The invention provides a retrieval method using a random quantization vocabulary tree and an image retrieval method based on the same, aiming at solving the defects in the prior art.

In order to achieve the technical purpose, the invention adopts the following technical scheme:

a retrieval method using a randomly quantized lexical tree, comprising the steps of:

(1) generating a nearest neighbor search tree, taking all the feature vectors of the whole database as root nodes of a first section, and segmenting downwards;

(2) in the second stage, k points are randomly selected from the whole database to serve as cluster centers, then each feature vector is distributed to the cluster center closest to the feature vector according to the selected similarity measurement method, the whole database is divided into k subsets, and downward segmentation is continued;

(3) in the third level, for each k clusters obtained from the second level, k feature points are randomly selected from the feature vector pool of the k clusters as the clustering centers of the next level, and then each feature point is subjected to similarity measurementThe eigenvector is assigned to the cluster center closest to it, forming k on the third level²Clustering;

(4) repeating the steps (2) and (3) until the feature vectors contained in all leaf nodes belong to the same class of objects or the number of the feature vectors contained in the leaf nodes is lower than a certain limit; where each feature vector has a class label associated with it.

In step (2), if the distances from the feature vector to the centers of two or more clusters are equal, one cluster is randomly selected.

In step (3), a new feature vector selected from the feature vector pool is allocated to the cluster center closest to the new feature vector, and when a leaf node of the allocated cluster center is reached, if all feature vector points in the leaf node have the same class label, the associated class label is allocated to the new feature vector, and then the operation is stopped; otherwise, searching in the distribution cluster again, selecting the feature vector with the shortest distance to the new feature vector in the cluster, distributing the class label associated with the feature vector to the new feature vector, and then stopping the operation.

An image retrieval method based on a retrieval method of a random quantized vocabulary tree comprises the following steps:

(1) firstly, dividing an image into a plurality of mutually overlapped sub-areas by an overlapping block dividing method;

(2) combining the characteristic information block of the image with the semantic characteristics of the image; because each extracted feature vector corresponds to one point of the feature space, all feature vectors in the feature vector library are divided into a plurality of patterns by performing non-guided learning (namely clustering in data mining) on the feature points in the feature space, so that the patterns in the same class have more similarity, and the patterns in different classes have larger dissimilarity; marking the feature points by class labels, wherein each class label has specific semantic information, combining the image feature information block with the semantic features, and interpreting different areas of the image to establish an image knowledge base;

(3) generating a nearest neighbor search tree, taking all image feature vectors in an image knowledge base as root nodes of a first section, and segmenting downwards;

(4) in the second stage, randomly selecting k points from an image knowledge base as cluster centers, then distributing each image feature vector to the cluster center closest to the image feature vector according to the selected similarity measurement method, dividing the whole database into k subsets, and continuously segmenting downwards;

(5) in the third level, for each k clusters obtained from the second level, k feature points are randomly selected from the feature vector pool of the k clusters as the cluster centers of the next level, and then each image feature vector is distributed to the cluster center nearest to the image feature vector by using a similarity measurement method, so that k clusters are formed on the third level²Clustering;

(6) repeating the steps (2) and (3) until the image feature vectors contained in all leaf nodes belong to the same class of objects or the number of the image feature vectors contained in the leaf nodes is lower than a certain limit; where each image feature vector has a class label associated with it.

In step (1), the overlap-and-block method divides an image with height × weight into N × N windows, shifts the row and column directions by Nhop pixels, and divides the image into a plurality of mutually overlapping sub-regions, so that a sufficiently small object contained in the image can be detected, reduces the size of the block window, increases the number of patches, and combines the color histogram with the spatial distribution of colors by mutually overlapping the sub-regions.

In the step (2), in the process of establishing the image knowledge base, a Chameleon clustering algorithm and an MST-based clustering algorithm are used for clustering the color histogram feature vector base of the image, class labels are set for clustering results, and the knowledge base based on the color histogram features is established.

By adopting the technical scheme, the method has the following beneficial effects:

(1) the image retrieval method of the invention overcomes the problem that the establishment of the vocabulary tree in the prior art needs a large amount of time, can establish the vocabulary tree in a short time and meet the real-time requirement;

(2) the invention uses the overlapping blocking method to refine the image into a plurality of blocks to extract the color histogram of the image as the feature vector library, effectively combines the color histogram of the image with the color space information, and overcomes the problem that the spatial characteristic of the color is ignored during the image feature extraction in the prior art;

(3) the method can extract the picture characteristics more quickly, meet the real-time requirement, simultaneously carry out non-guided learning on the characteristic database, and mark different areas of the scene image through the area mark to form the knowledge base.

Drawings

FIG. 1 is a schematic view of the present invention;

FIG. 2 is a schematic illustration of the overlapping block method of the present invention;

FIG. 3 is a clustering result of 10648 dimensional RGB histograms of 21-14;

FIG. 4 is a clustering result of 5000-dimensional HSV histograms of 24-16;

FIG. 5 is the clustering results of the 5832-dimensional Opponent histogram of FIGS. 24-16;

FIG. 6 is the clustering results of the 10648 dimensional Transformed histogram of FIGS. 21-14;

FIG. 7 is a comparison graph of RGB histogram accuracy for groups 21-14;

FIG. 8 is a graph of RGB histogram accuracy comparison of the 24-16 sets;

FIG. 9 is a comparison graph of the accuracy of 27-18 sets of RGB histograms.

DETAILED DESCRIPTION OF EMBODIMENT (S) OF INVENTION

The present invention is further described with reference to the following drawings and embodiments.

As shown in fig. 1, the searching method using the randomly quantized vocabulary tree includes the following steps:

(3) in the third stageFor each k clusters obtained from the second level, k feature points are randomly selected from the feature vector pool of the k clusters as the cluster centers of the next level, and then each feature vector is distributed to the cluster center nearest to the feature point by using a similarity measurement method, so that k clusters are formed on the third level²Clustering;

For a large database, the random quantization tree search method is selected to follow the thought of a vocabulary tree, but an important improvement is made on the basis of the vocabulary tree search method, and a nearest neighbor search tree is generated. As shown in fig. 1, given a database, there is only one node in the first level, containing all the feature vectors, which becomes the root node. And in the second stage, randomly selecting K points from the whole database as the centers of the clusters, then distributing each feature vector to the nearest cluster center according to the selected similarity measurement method, and dividing the whole database into K subsets. At the third level, for each K clusters obtained from the second level, K feature points are randomly selected from the feature vector pool of the K clusters as the cluster centers of the next level, and then each feature vector is allocated to the cluster center closest to the feature point by using a similarity measurement method, so that K clusters are formed on the third level. This process continues until all leaf nodes contain feature vectors that belong to the same class of objects (i.e., the node is clean) or until the number of leaf nodes contain feature vectors below a certain limit (e.g., 50). Each feature vector has a class label associated with it.

In branching the tree, the distance of each data item to other data items can be updated to a smaller value by a nearest neighbor search tree. This strategy ensures that the closest data points in space are more likely to be assigned to the same partition. However, because any one data point in a partition is closest to its own cluster center (not the nearest neighbor) compared to the centers of the other partitions, a data point is randomly selected from a cluster if the data points are equidistant from the centers of two or more clusters.

A given new feature vector is searched through a random quantization tree. The new feature vector is calculated at each level along a particular path of the random quantization tree for the K cluster centers, resulting in a cluster center point for the new feature vector that is closest to the K cluster center points. When a leaf node is reached, if the leaf node is clean (i.e., all feature vector points in the leaf node have the same class label), the associated class label is assigned to the new feature vector, and then the operation is stopped. Otherwise, a nearest neighbor search is carried out in the vectors of the related clusters, the search result is the feature vector with the shortest distance obtained according to the selected similarity measurement method, the class label associated with the feature vector is distributed to a new feature vector, and then the operation is stopped.

As shown in FIG. 2, in step (1), the overlap-blocking method divides the image with height × weight into N × N windows, shifts the rows and columns by Nhop pixels, and divides the image into several sub-regions overlapping each other, so that the object contained in the image can be detected out with small enough size, reduces the size of the window of the block, increases the number of patches, combines the color histogram with the spatial distribution of the color by overlapping the sub-regions, divides the image with size into a plurality of blocks by overlapping the sub-regions:

line number: blockrows ═ height-N)/Nhop +1

The number of columns: blockcols ═ N (weidth-N)/Nhop +1

The number of patches for each block is numofSamples × blocks

The size of the image is height × weight pixels, and the size of the generated processed image is determined by the size of the patch window, the number of patch blocks generated by the image is changed by the size of the patch window of the image, when the patch window is reduced, the number of patch blocks generated by processing the image is increased, and vice versa.

The color histogram of the image is simply extracted by blocking the image, and the image is only divided into blocks without any semantic information. Combining the feature information blocks of the image with their semantic features is the purpose of building an image knowledge base. Because each extracted feature vector corresponds to one point of the feature space, all feature vectors in the feature vector library are divided into a plurality of patterns by performing non-guided learning (namely clustering in data mining) on the feature points in the feature space, so that the patterns in the same class have more similarity, and the patterns in different classes have larger dissimilarity. And marking the feature points by class labels, wherein each class label has specific semantic information, and combining the image feature information block with the semantic features to explain different areas of the image.

The superiority of the present invention is further demonstrated by experimental means below.

Clustering of image feature data

Clustering is carried out on the characteristic vector database by adopting a chameleon clustering algorithm and a clustering algorithm based on MST respectively, objects in the image are divided into a plurality of clusters, and the clusters are marked to form a knowledge base. And the clustered database is represented by the color images. The feasibility of the image feature extraction method is verified by comparing the original image set, the MST-based clustering result image and the Chameleon clustering result image.

In the experiment, because the image data set has more images, only partial image clustering results in the database are displayed, and because the characteristic database has a plurality of groups, a group of clustering results of an RGB space, an HSV space, an Opponent space and a Transformed space are respectively displayed.

TABLE 1 Primary scenes and their semantic relationships in RGB knowledge base based on RGB color histogram

As shown in fig. 3, the clustering results of the 10648-dimensional RGB histograms are shown as 21-14.

TABLE 2 HSV color histogram-based HSV knowledge base primary scenarios and their semantic relationships

As shown in FIG. 4, there are 24-16 clustering results for the 5000-dimensional HSV histogram.

Table 3 Opponent color histogram based Opponent knowledge base primary scenes and their semantic relationships

As shown in FIG. 5, the clustering results of the 5832-dimensional Opponent histogram of 24-16 are shown.

Table 4 Main scenes and their semantic relationships in a transformated color histogram-based transformated knowledge base

As shown in FIG. 6, the clustering results of the 10648 dimensional Transformed histogram of FIGS. 21-14 are shown.

Extracting characteristic velocity

The size of the processed image is 50 × 90 pixels if the selected image size is 720 × 1280 pixels, the size of the patch window is 21 × 21, the value of the shifted pixel is set to 14 (shown as 21-14), the size of the processed image is 16 (shown as 24-16) if the patch window size is 24 × 24, and the value of the shifted pixel is set to 18 (shown as 27-18) if the patch window size is 27 × 27, the size of the processed image is 39 × 70.

By a more refined overlap blocking method, an image is divided into 4500 blocks, 3476 blocks, 2730 blocks, and a color histogram of each block is extracted. In the conventional image blocking, a gray level color histogram or an HSV color histogram of an image is mainly extracted, and herein, an RGB color histogram, an HSV color histogram, an Opponent color histogram and a Transformed color histogram of the image are extracted to establish an image feature vector library.

In the experiment, for 35 images with the size of 720 × 1280, three overlapping and blocking methods of 21-14, 24-16 and 27-18 are respectively adopted to extract 10000 dimensions of RGB color histograms, 10648 dimensions of HSV color histograms, 10000 dimensions of Opponent color histograms, 10000 dimensions of Transformed color histograms and Gabor texture features of the images, and SIFT feature points of the same images.

The experiment respectively counts the extraction time of different characteristics of 35 pictures, and the average value of the characteristic extraction of each picture is shown in the following table.

TABLE 5

Table 5 shows: the same image blocking method is adopted, and even if the high-dimensional color histogram of the image is extracted, the extraction speed of the method is obviously superior to that of the extraction method of Gabor texture features; meanwhile, the time spent on extracting the SIFT feature points of the whole image is much slower than the time spent on extracting the color histogram of the image after the image is partitioned. Therefore, the feature extraction method adopted by the text can quickly extract the image features.

Rate of accuracy

Since the nearest neighbor retrieval is that the retrieval object and the nearest neighbor belong to the same class, the retrieval accuracy is highest. We use nearest neighbor as the standard search result set. And comparing the retrieval result of the vocabulary tree, the retrieval result of the random quantization tree and the nearest neighbor retrieval result, and analyzing the retrieval accuracy of the two trees.

RGB histogram accuracy contrast

First, we search for different sets (21-14, 24-16, 27-18) of multiple dimensions (64 dimensions, 125 dimensions, 216 dimensions, 512 dimensions, 1000 dimensions, 2744 dimensions, 5832 dimensions, 10648 dimensions) in the RGB color space, and compare the results with table 2, table 3, and table 4, where KQtree is a randomly quantized vocabulary tree and VTree is a conventional vocabulary tree.

Comparison result of RGB histogram accuracy: it is shown from fig. 7, 8 and 9 that the accuracy of the random quantization tree is significantly higher than that of the vocabulary tree.

In fig. 7, the accuracy of the random quantization tree is highest at 1000 dimensions, reaching 83.03%, and it is shown that the search result of the high dimension is better than the search result of the low dimension. In fig. 8, the accuracy of the random quantization tree is the highest at 5832 dimensions, which reaches 86.73%, and it is also shown that the search results are better for the high dimensions than for the low dimensions. In fig. 9, the random quantization tree has the highest accuracy in 512 dimensions, which reaches 85.49%, and the high-dimensional data retrieval effect is better than the low-dimensional data retrieval effect, but is not obvious, and the middle-dimensional retrieval effect is the best.

Fig. 7, 8, and 9 are compared together, and it is found that the overall search effect of the RGB color histogram in the high-dimensional data space is better than that in the low-dimensional data space, and meanwhile, for different sizes of the patch windows, the smaller the window is, the more the patch numbers are, the richer the image information is, but the more patch is, the higher the search accuracy of the image is, the different dimensions have different expressions, and there is no uniform rule.

Establishing lexical tree time

The runtime of multiple dimensions (64 dimensions, 125 dimensions, 216 dimensions, 512 dimensions, 1000 dimensions, 2744 dimensions, 5832 dimensions, 10648 dimensions) of different groups (21-14, 24-16, 27-18) in the RGB histogram is represented by table 6, table 7, and table 8, where unit is second, where KQtree is a randomly quantized vocabulary tree and VTree is a conventional vocabulary tree.

TABLE 6 Window 21 × 21, Shift 14, randomly quantizing the lexical tree and lexical tree runs times in different dimensions of the RGB histogram

TABLE 7 Window 24 × 24, Shift 16, randomly quantizing the lexical tree and lexical tree runtime in different dimensions of the RGB histogram

TABLE 8 Window 27 × 27, Shift 18, randomly quantizing the lexical tree and lexical tree runs times in different dimensions of the RGB histogram

In the RGB histogram, the running speed of the random quantization tree is significantly faster than the running speed of the lexical tree. As the RGB histogram dimension increases, the vocabulary tree runtime increases in geometric multiples of the random quantization tree runtime.

The runtime in seconds for the multiple dimensions (64D, 125D, 216D, 512D, 1000D, 2744D, 5832D, 10648D) of different groups (21-14, 24-16, 27-18) of the lexical tree and lexical tree in the Opponent histogram is shown in tables 9, 10, 11.

TABLE 9 Window 21 × 21, Shift 14, randomly quantizing the lexical tree and lexical tree run times in different dimensions of the Opponent histogram

TABLE 10 Window 24 × 24, Shift 16, randomly quantizing the lexical tree and lexical tree run times in different dimensions of the Opponent histogram

TABLE 11 Window 27 × 27, Shift 18, randomly quantizing the lexical tree and lexical tree run times in different dimensions of the Opponent histogram

In the Opponent histogram, the random quantization tree runs significantly faster than the lexical tree. As the Opponent histogram dimension increases, the vocabulary tree run time increases geometrically in multiples of the random quantization tree run time.

In conclusion, the retrieval method of the random quantization vocabulary tree is obviously superior to the vocabulary tree in time efficiency.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and additions can be made without departing from the principle of the present invention, and these should also be considered as the protection scope of the present invention.

Claims

1. An image retrieval method based on a retrieval method of a random quantized vocabulary tree, characterized in that the retrieval method using the random quantized vocabulary tree comprises the following steps:

(2) in the second stage, k points are randomly selected from the whole database to serve as cluster centers, then each feature vector is distributed to the cluster center closest to the feature vector according to the selected similarity measurement method, the whole database is divided into k subsets, and downward segmentation is continued; if the distances from the feature vector to the centers of two or more clusters are equal, one cluster is randomly selected;

(3) in the third level, for each k clusters obtained from the second level, k feature points are randomly selected from the feature vector pool of the k clusters as the cluster centers of the next level, and then each feature vector is distributed to the cluster center nearest to the feature point by using a similarity measurement method, so that k clusters are formed on the third level²Clustering; when a leaf node of the distributed cluster center is reached, if all feature vector points in the leaf node have the same class label, distributing the associated class label to a new feature vector, and then stopping operation; otherwise, searching in the distribution cluster again, selecting the feature vector with the shortest distance from the new feature vector in the cluster, distributing the class label associated with the feature vector to the new feature vector, and then stopping operation;

(4) repeating the steps (2) and (3) until the feature vectors contained in all leaf nodes belong to the same class of objects or the number of the feature vectors contained in the leaf nodes is lower than a certain limit; wherein each feature vector has a class label associated with it;

the image retrieval method comprises the following steps:

(1) the method comprises the steps of firstly, dividing an image into a plurality of mutually overlapped sub-regions by an overlapping blocking method, specifically, dividing the image with height × weight into a plurality of mutually overlapped sub-regions by using an N × N window, shifting the row direction and the column direction according to Nhop pixels, and combining a color histogram with the spatial distribution of colors by mutually overlapping sub-regions in order to detect a small enough object contained in the image, reducing the size of a block window and increasing the number of patches;

(2) combining the characteristic information block of the image with the semantic characteristics of the image; because each extracted feature vector corresponds to one point of the feature space, all feature vectors in the feature vector library are divided into a plurality of patterns by performing non-guided learning (namely clustering in data mining) on the feature points in the feature space, so that the patterns in the same class have more similarity, and the patterns in different classes have larger dissimilarity; marking the feature points by class labels, wherein each class label has specific semantic information, combining the image feature information block with the semantic features, and interpreting different areas of the image to establish an image knowledge base; specifically, in the process of establishing an image knowledge base, a Chameleon clustering algorithm and an MST-based clustering algorithm are used for clustering a color histogram feature vector base of an image, class labels are set for clustering results, and the knowledge base based on color histogram features is established;