CN107656989B - Neighbor query method based on data distribution awareness in cloud storage system - Google Patents

Abstract

The invention discloses a neighbor query method based on data distribution perception in a cloud storage system. The method uses the principal component of the data as the projection vector of the local sensitive hash, and further quantifies the weight and sum of each hash function in the index table. Adjust the cutting interval of the hash function in each hash table to ensure the accuracy of the neighbor query and reduce the number of hash tables required to build the index, thereby reducing the space overhead of the hash table. Furthermore, the method refines the query result set according to the hash collision frequency of the neighbor query results, eliminates a large number of irrelevant elements, greatly reduces the amount of data used for distance calculation, and reduces the query delay. The present invention can Make full use of the characteristics of data distribution, meet the characteristics of fast query, and have good scalability.

Description

Neighbor query method based on data distribution awareness in cloud storage system

technical field

The invention belongs to the technical field of computer storage, and more specifically relates to a neighbor query method based on data distribution perception in a cloud storage system.

Background technique

Massive storage systems consume a lot of system resources, such as computing, storage, and network, to support query-related requests. However, real-time processing and analysis of massive high-dimensional data is still a huge challenge. For query requests, obtaining accurate results is very time-consuming and difficult to operate, because users often have difficulty providing accurate descriptions for query requests. Therefore, the nearest neighbor query service has received more and more attention in practical applications due to its real-time nature.

Locality Sensitive Hashing (LSH) has the characteristics of simple hash calculation and the ability to maintain data locality, and is widely used to support neighbor query services.

The existing LSH-based neighbor query methods have the following problems:

(1) The accuracy is low. Traditional LSH-based methods randomly select the projection vector of the hash function without considering the data distribution. Uniformly distributed data is mapped to the hash table with equal probability through random projection vectors, so the data in the hash bucket is balanced. However, in practical applications, the data distribution is mostly uneven. As a result, the unevenly distributed data is mapped through randomly selected projection directions, causing many irrelevant data to be aggregated together, reducing the accuracy of the nearest neighbor query operation.

(2) The space efficiency is low. The mapping vectors of conventional LSH-based hash functions are independent of the data distribution. The traditional LSH method utilizes a large number of hash tables to ensure query accuracy. Therefore, severe memory overhead becomes the performance bottleneck of traditional LSH-based methods.

(3) The query delay is large. Due to the random mapping of traditional LSH methods, many irrelevant data in the query operation are detected and stored into the query result set. Therefore, the result set contains a large number of candidate elements and the distance calculation between the query element and the query element is required, and this calculation is very time-consuming, and the query delay is too large for the user.

Contents of the invention

Aiming at the above defects or improvement needs of the prior art, the present invention provides a neighbor query method based on data distribution perception in a cloud storage system, thereby solving the problem of low precision and low space efficiency in the neighbor query in a mass storage system And the technical problem of long query delay.

In order to achieve the above object, the present invention provides a neighbor query method based on data distribution perception in a cloud storage system, including:

S1. Randomly extract some data from the original high-dimensional data set to form a high-dimensional feature data set;

S2. Express each element in the high-dimensional feature data set as a multi-dimensional vector, so as to represent the high-dimensional feature data set as a matrix composed of multiple multi-dimensional vectors, and calculate the covariance matrix of the matrix offline through principal component analysis, Then get the eigenvectors and eigenvalues of the matrix;

S3. Obtain the required number of hash tables in the index table, the number of hash functions in each hash table, and the collision threshold;

S4. According to the descending order of the eigenvalues, the eigenvectors corresponding to each eigenvalue are one-to-one corresponded as the projection vector of the hash function, and the calculation of each hash function in each hash table is performed according to the eigenvalues corresponding to the eigenvectors. Weight, then adjust the cutting interval of the hash function in each hash table, and finally map the hash function obtained by optimizing the original high-dimensional data set to the entire index table, and store the elements that generate hash conflicts through the linked list;

S5. For each query point, in each hash table, the corresponding hash value is calculated through the optimized hash function, and the hash value is used to locate the location where the hash conflict occurs in the hash table, and the All elements in the linked list of the position are stored in the result candidate set, and the number of hash collisions between each element in the result candidate set and the query point is recorded, and the elements smaller than the preset conflict threshold are removed to obtain the neighbor query set. Each point in the query set is compared with the query point for distance calculation, and all elements whose distance from the query point is less than the preset distance threshold are output.

Preferably, step S2 specifically includes the following sub-steps:

S2.1. Treat n elements in the high-dimensional feature data set X as n vectors containing d variables, then express the high-dimensional feature data set X as a matrix composed of n d-dimensional vectors:

S2.2, by Get the covariance matrix, where, represents the variance, while the covariance is According to the calculation of the covariance matrix S, the eigenvector group V and the eigenvalue group N are obtained;

S2.3. The eigenvectors corresponding to the first k×L eigenvalues with larger eigenvalues in the eigenvalue group N are used as the principal component group V' of the high-dimensional feature data set X, and the high-dimensional feature data set is mapped by V' X is represented as a data set Y, and Y=XV', where k represents the number of hash functions in each hash table, and L represents the number of required hash tables in the index table.

Preferably, step S3 specifically includes the following sub-steps:

S3.1, by Get the number L of hash tables in the index table, and get the number k of hash functions in each hash table, where p ₁ represents the probability that two points are approximate points and a hash collision occurs, and p ₂ represents two The point is not an approximate point but the probability of hash collision, α is the collision ratio threshold and p ₂ <α<p ₁ , δ is the success rate of neighbor query, β is the misjudgment rate of local sensitive hash LSH;

S3.2. Obtain the conflict ratio threshold α corresponding to the minimum value of the hash table number L, and in

S3.3, obtained from the value of α

S3.4. Obtain the conflict threshold according to the values of α and L'

Preferably, step S4 specifically includes the following sub-steps:

S4.1, express the LSH function as: Where a is the projection vector, p is any point in the multidimensional space in the high-dimensional feature data set X, b is a randomly selected real number in the range [0, ω), and ω is the projection cutting interval;

S4.2. Correspond the k×L eigenvectors selected in step S2.3 one by one as the projection vector of each hash function, assuming that the eigenvalues corresponding to the k eigenvectors in each hash table are according to The descending order is N=[n ₁ ,n ₂ ,...,n _k ], then the weight of each hash function is: Then in each hash table, the hash value of point p is

S4.3. In each hash table, the cutting interval ω of the k hash functions is the same, and the interval ω of the hash functions in the next hash table is half of the interval ω of the hash functions in the previous hash table ;

S4.4. According to the projection vector and cutting interval of the hash function in each hash table, L hash tables are constructed, each hash table contains k hash functions, and the high-dimensional feature data set X All the points in the multidimensional space of are inserted into each hash table in the index table through the hash map, and the points where the hash collision occurs are stored through the linked list.

Preferably, step S5 specifically includes the following sub-steps:

S5.1. For the query point q, calculate its hash value g _i (q) (1≤i≤L) in each hash table, and all elements in the corresponding hash bucket list that will cause a hash collision All are saved in the query result set C(q), and the repeated elements are only saved once, and the approximate set of the query point q in the high-dimensional feature data set X is obtained;

S5.2. Record the number of times that each point in the query result set C(q) collides with the query point q in the index table, and for any point p in the query result set C(q), the number of times it collides with the query point q The number of conflicts is: Assuming that the conflict threshold is m, that is, when the number of collisions between a point in the query result set C(q) and the query point q is greater than m, the point is considered to be similar to the query point q, and the point is stored in the refined result set C'(q);

S5.3. For all the points in the refined result set C'(q), calculate the Euclidean distance with the query point q in turn. When the distance between the two points is less than the preset distance threshold, use this point as the query point q approximation point.

Generally speaking, compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

This method uses the principal component of the data as the projection vector of local sensitive hashing, and further quantifies the weight of each hash function in the index table and adjusts the cutting interval of the hash function in each hash table to ensure that the nearest neighbor query Accurate while reducing the number of hash tables required to build the index, thereby reducing the space overhead of the hash table. Furthermore, the method refines the query result set according to the hash collision frequency of the neighbor query results, eliminates a large number of irrelevant elements, greatly reduces the amount of data used for distance calculation, and reduces the query delay.

Description of drawings

Fig. 1 is a schematic flow diagram of a neighbor query method based on data distribution awareness in a cloud storage system provided by an example of the present invention;

Fig. 2 is a schematic flow chart of a principal component analysis calculation method provided by an example of the present invention;

Fig. 3 is a schematic flow chart of a parameter setting method provided by an example of the present invention;

Fig. 4 is a schematic flow chart of a method for constructing an index table provided by an example of the present invention;

Fig. 5 is a schematic flowchart of a method for neighbor query provided by an example of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The present invention is a neighbor query method based on data distribution perception in a cloud storage system, which takes into account the characteristics of data distribution and uses principal component analysis to guide the selection of projection vectors in the LSH algorithm, and further quantifies the weight of each hash function in the index table Value and adjust the cutting interval size of the hash function in each hash table to ensure the accuracy of the neighbor query and reduce the number of hash tables required to build the index, thereby reducing the space overhead of the hash table. Furthermore, the method refines the query result set according to the hash collision frequency of the neighbor query results, eliminates a large number of irrelevant elements, greatly reduces the amount of data used for distance calculation, and reduces the query delay.

As shown in Figure 1, it is a schematic flow diagram of a neighbor query method based on data distribution awareness in a cloud storage system provided by an example of the present invention; the method shown in Figure 1 includes the following steps:

S1. Data set sampling: Randomly extract some data from the original high-dimensional data set to form a high-dimensional feature data set to ensure time efficiency;

S2. Principal component analysis calculation: Perform principal component analysis on the high-dimensional feature data set obtained in step S1 to obtain the projection vector of the hash function, specifically: express each element in the high-dimensional feature data set as a multi-dimensional vector, as Express the high-dimensional feature data set as a matrix composed of multiple multi-dimensional vectors, and calculate the covariance matrix of the matrix offline through principal component analysis, and then obtain the eigenvectors and eigenvalues of the matrix;

In an optional implementation manner, as shown in FIG. 2, step S2 specifically includes the following sub-steps:

S2.1. Treat n elements in the high-dimensional feature data set X as n vectors containing d variables, then express the high-dimensional feature data set X as a matrix composed of n d-dimensional vectors:

S2.2, by Get the covariance matrix, where, represents the variance, while the covariance is According to the calculation of the covariance matrix S, the eigenvector group V and the eigenvalue group N are obtained;

S2.3. The eigenvectors corresponding to the first k×L eigenvalues with larger eigenvalues in the eigenvalue group N are used as the principal component group V' of the high-dimensional feature data set X, and the high-dimensional feature data set is mapped by V' X is represented as a data set Y, and Y=XV', where k represents the number of hash functions in each hash table, and L represents the number of required hash tables in the index table.

Generally, the data variance mapped by the eigenvectors corresponding to the first few larger eigenvalues represents a larger amount of data, which can better reflect the query performance. Furthermore, the first few eigenvectors that can reflect the data distribution are used as the projection vectors in LSH to construct the index table to reduce space overhead.

S3. Parameter setting: obtain the required number of hash tables in the index table, the number of hash functions in each hash table, and the collision threshold;

In an optional implementation manner, as shown in FIG. 3, step S3 specifically includes the following sub-steps:

S3.1, by Get the number L of hash tables in the index table, and get the number k of hash functions in each hash table, where p ₁ represents the probability that two points are approximate points and a hash collision occurs, and p ₂ represents two The point is not an approximate point but the probability of hash collision, α is the collision ratio threshold and p ₂ <α<p ₁ , δ is the success rate of neighbor query, β is the misjudgment rate of local sensitive hash LSH;

Among them, the preferred value of δ is β is preferably

S3.2. Obtain the conflict ratio threshold α corresponding to the minimum value of the hash table number L, and in

Specifically, let and Then L=max(L ₁ , L ₂ ), since p ₂ <α<p ₁ , then L ₁ increases with the increase of α, and L ₂ decreases with the increase of α. When L ₁ =L ₂ , the value of the number L of hash tables reaches the minimum, and we get in

S3.3. Bring the value of α into L ₁ to get

S3.4. Obtain the conflict threshold according to the values of α and L'

S4. Index table construction: According to the descending order of the eigenvalues, the eigenvectors corresponding to each eigenvalue are used as the projection vector of the hash function one by one, and each hash table in each hash table is calculated according to the eigenvalues corresponding to the eigenvectors. The weight of the hash function, and then adjust the cutting interval size of the hash function in each hash table, and finally map the hash function obtained by optimizing the original high-dimensional data set to the entire index table, and generate elements of hash conflicts Stored through a linked list;

In an optional implementation manner, as shown in FIG. 4, step S4 specifically includes the following sub-steps:

S4.1. Weight quantization: express the LSH function as: Where a is the projection vector, p is any point in the multidimensional space in the high-dimensional feature data set X, b is a randomly selected real number in the range [0, ω), and ω is the projection cutting interval;

Wherein, generally k hash functions are used in each hash table to obtain the hash value of the point in the multi-dimensional space. For the traditional LSH based on random mapping, each hash table needs to assign a random weight to each hash function, and then calculate the weighted sum as the hash value of the point. For any point p in the set X, its hash value in a certain hash table can be calculated as: g(p)=a ₁ *h ₁ (p)+a ₂ *h ₂ (p)+.. .+a _k *h _k (p), where the weight a _i is a random number from 0 to 1, subject to p stable distribution. A hash function with a better projection vector can separate the aggregated data through mapping calculations, that is, the higher probability maps similar points to the same hash bucket, and the smaller probability will be farther away The point hash maps to the same hash bucket. Intuitively, hash functions with better projection vectors should be assigned larger weights for better query performance.

S4.2. Correspond the k×L eigenvectors selected in step S2.3 one by one as the projection vector of each hash function, assuming that the eigenvalues corresponding to the k eigenvectors in each hash table are according to The descending order is N=[n ₁ ,n ₂ ,...,n _k ], then the weight of each hash function is: Then in each hash table, the hash value of point p is

S4.3. Cutting interval adjustment: In each hash table, the cutting interval ω of the k hash functions is the same, and the interval ω of the hash functions in the next hash table is the same as that of the hash functions in the previous hash table. half of the interval ω;

Specifically, the parameter ω reflects the granularity of hash collisions. A larger interval ω can increase the probability of hash collisions at similar points, but it may also map distant points to the same hash bucket, which will affect the query accuracy. A smaller interval ω can reduce the probability of hash collisions of distant points, but it may also cause similar points to be mapped to different hash buckets, which affects the query recall rate. In the embodiment of the present invention, the value of the cutting interval ω is adjusted to improve query performance. In each hash table, the cutting interval ω of the k hash functions is the same. The interval ω of the hash function in the next hash table is half of the interval ω of the hash function in the previous hash table, that is to say, for the i-th hash table, the size of the interval ω in the hash function is: Where ω ₀ is the initial interval value of the hash function of the first hash table.

S4.4. Construction insertion: according to the projection vector and cutting interval of the hash function in each hash table, L hash tables are constructed, each hash table contains k hash functions, and the high-dimensional feature data All multi-dimensional space points in the set X are inserted into each hash table in the index table through the hash map, and the points where the hash collision occurs are stored through the linked list.

S5. Neighbor query: For each query point, in each hash table, calculate the corresponding hash value through the optimized hash function, and locate the location where the hash conflict occurs in the hash table through the hash value , store all the elements in the linked list at this position in the result candidate set, record the number of hash collisions between each element in the result candidate set and the query point, remove the elements smaller than the preset conflict threshold, and obtain the neighbor query set, By comparing each point in the neighbor query set with the query point for distance calculation, all elements whose distance to the query point is less than the preset distance threshold are output.

In an optional implementation manner, as shown in FIG. 5, step S5 specifically includes the following sub-steps:

S5.1. For the query point q, calculate its hash value g _i (q) (1≤i≤L) in each hash table, and all elements in the corresponding hash bucket list that will cause a hash collision All are saved in the query result set C(q), and the repeated elements are only saved once, and the approximate set of the query point q in the high-dimensional feature data set X is obtained;

S5.2. Record the number of times that each point in the query result set C(q) collides with the query point q in the index table, and for any point p in the query result set C(q), the number of times it collides with the query point q The number of conflicts is: Assuming that the conflict threshold is m, that is, when the number of collisions between a point in the query result set C(q) and the query point q is greater than m, the point is considered to be similar to the query point q, and the point is stored in the refined result set C'(q);

S5.3. For all the points in the refined result set C'(q), calculate the Euclidean distance with the query point q in turn. When the distance between the two points is less than the preset distance threshold, use this point as the query point q approximation point.

Wherein, the preset distance threshold may be determined according to actual needs.

Those skilled in the art can easily understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (5)

1. the nearest Neighbor based on data distribution perception in a kind of cloud storage system characterized by comprising

S1, partial data composition high dimensional feature data set is randomly selected from original high dimensional data concentration；

S2, by each of high dimensional feature data set element representation be a multi-C vector, by high dimensional feature data set table It is shown as the matrix being made of multiple multi-C vectors, by principal component analysis come the covariance matrix of the off-line calculation matrix, in turn Obtain the feature vector and characteristic value of the matrix；

S3, required Hash table number in concordance list, hash function number and conflict threshold in each Hash table are obtained；

S4, the descending order according to characteristic value regard the corresponding feature vector of each characteristic value as hash function correspondingly Projection vector, and the weight of each hash function in each Hash table is calculated according to the corresponding characteristic value of feature vector, then adjust The cutting gap size of hash function in whole each Hash table, the Hash letter for finally obtaining original High Dimensional Data Set by optimization Number is mapped in entire concordance list, and the element for generating hash-collision passes through storage of linked list；

S5, for each query point, in each Hash table, by optimization after hash function corresponding Hash is calculated Value is navigated in Hash table the position that hash-collision occurs by cryptographic Hash, all elements in the chained list of the position is all stored in As a result in candidate collection, the number that each element and query point in result candidate collection generate hash-collision is recorded, will be less than pre- If the element of conflict threshold removes, NN Query set is obtained, by clicking through each point in neighbour's query set with inquiry Row distance calculating is compared, and all elements for being less than pre-determined distance threshold value with query point distance are exported.

2. the method according to claim 1, wherein step S2 specifically includes following sub-step:

S2.1, n element in high dimensional feature data set X is regarded to the n vectors comprising d variable as, then by high dimensional feature number The matrix being made of n d dimensional vector is expressed as according to collection X:

S2.2, byObtain covariance matrix, whereinIndicate variance, together When covariance beFeature vector group V is calculated according to covariance matrix S With eigenvalue cluster N；

S2.3, using the corresponding feature vector of the biggish preceding k × L characteristic value of characteristic value in eigenvalue cluster N as high dimensional feature number According to the principal component group V' of collection X, is mapped by V' and high dimensional feature data set X is expressed as data set Y, and Y=XV', wherein k table Show the number of hash function in each Hash table, L indicates required Hash table number in concordance list.

3. according to the method described in claim 2, it is characterized in that, step S3 specifically includes following sub-step:

S3.1, byHash table number L in concordance list is obtained, and The number k of hash function in each Hash table is obtained, wherein p₁Indicate that two points are the probability of approximate point and generation hash-collision, p₂Indicate that two points are not approximate point but the probability that hash-collision occurs, α is conflict proportion threshold value and p₂< α < p₁, δ is neighbour The success rate size of inquiry, β are the False Rate of the sensitive Hash LSH in part；

S3.2, corresponding conflict proportion threshold value α when Hash table number L value minimum is obtained, andWherein

S3.3, it is obtained by the value of α

S3.4, according to the value of α and L', obtain conflict threshold

4. according to the method described in claim 3, it is characterized in that, step S4 specifically includes following sub-step:

S4.1, by LSH function representation are as follows:Wherein a is projection vector, and p is more in high dimensional feature data set X Any point in dimension space, b be range [0, ω) in a randomly selected real number, ω be projection cutting interval；

S4.2, k × L feature vector for selecting step S2.3 sequentially correspondingly as the projection of each hash function to Amount, it is assumed that the corresponding characteristic value of k feature vector in each Hash table is followed successively by N=[n according to descending arrangement₁,n₂,..., n_k], then the weight of each hash function are as follows:1≤i≤k, and then in each Hash table, the cryptographic Hash of point p For

S4.3, in each Hash table, the cutting interval ω of k hash function is identical, hash function in next Hash table Interval ω is the half of the interval ω of hash function in a upper Hash table；

S4.4, it is spaced according to the projection vector of hash function in each Hash table and cutting, to construct L Hash table, each Kazakhstan Uncommon table all includes k hash function, and the point of all hyperspace in high dimensional feature data set X is all inserted by Hash mapping In each Hash table into concordance list, the point that hash-collision occurs passes through storage of linked list.

5. according to the method described in claim 4, it is characterized in that, step S5 specifically includes following sub-step:

S5.1, for query point q, calculate its cryptographic Hash g in each Hash table_i(q), hash-collision will occur for 1≤i≤L Correspondence Hash barrel chain table in all elements be all saved in query results C (q), repeat element only saves once, obtains Approximate set of the query point q in high dimensional feature data set X；

In S5.2, record queries result set C (q), the number that each point and query point q are clashed in concordance list, and for Any point p in query results C (q), the conflict number of it and query point q are as follows:Assuming that conflict threshold is m, i.e. certain point and inquiry in query results C (q) When the conflict number of point q is greater than m, just think that the point is approximate with query point q, and the point be stored in refining result set C'(q) in；

S5.3, for refine result set C'(q) in all the points, successively with query point q carry out Euclidean distance calculating, work as point-to-point transmission Distance be less than pre-determined distance threshold value when, then using the point as the approximate point of query point q.