CN106055674A - top-k arrangement query method based on metric space in distributed environment - Google Patents

Abstract

The invention discloses a top-k domination query method based on a metric space in a distributed environment, which includes the following steps in turn: Step 1: Given a query input set Q and a distance formula d() in the metric space, the distance formula is used to measure The distance between the entire data object and the query object Q; Step 2: According to Step 1, a parallel algorithm based on ensemble ANN and k-skyband is proposed. By making full use of the characteristics of parallel computing between nodes in a distributed environment, pruning and sorting greatly improve the performance of top-k dominated query based on metric space in a large data set environment, and speed up the query. User's decision to provide services.

Description

A Top-k Domination Query Method Based on Metric Space in Distributed Environment

technical field

The invention relates to a query method, in particular to a parallel top-k dominance query method based on metric space in a distributed environment of mass data concentration.

Background technique

As an important complex query, the top-k domination query based on metric space is getting more and more attention. It returns a part of the data that meets the user's needs from the massive multi-dimensional data set. This type of query provides decision-making for users, for example, it has a wide range of applications in web search, multimedia retrieval, e-commerce and other fields. This query does not require the user to give an evaluation function and the result set is controllable. It calculates the dominance score of each object and returns the k result sets with the highest dominance scores.

The top-k domination query based on metric space is defined as follows: O={o ₁ ,o ₂ ,…,o _n } represents the set of all data objects, o _i represents the i-th data object, and each data object has D dimension, and is a point in space. For a top-k dominant query in a metric space, Q represents the query input set, and d() represents the distance formula in the metric space. This distance formula can be defined by yourself, such as the shortest path in the graph, the maximum flow in the network, and the Manhattan distance. etc., k means return the k results with the highest dominance score. The meaning of dominance is: if there is o _i ∈ O, o _i' ∈ O, use the symbol < to indicate the dominance relationship between two objects, if o _i <o _i' , then:

Given a data object o _i ∈ O, the dominance score dscore of the object o _i is the number of objects dominated by it in the entire data set, as follows:

dscore＝|{o _j ∈ O|o _i ＜o _j }|

The top-k domination query based on the metric space finally only needs to get the k elements with the highest domination score, which is a dynamic top-k domination query. Tiakas E and others first proposed this concept, but it was only studied in the traditional stand-alone mode. At present, with the rapid increase of data sets, the traditional stand-alone algorithm encounters performance bottlenecks, and Tiakas E and others use M-tree. The index storage structure is completely unsuitable for large data sets, which will lead to a large amount of data redundancy, so it is imminent to study the parallel top-k dominance algorithm based on metric space.

Contents of the invention

Purpose of the invention: the purpose of the present invention is to solve the deficiencies in the prior art, and provide a parallel top-k dominance query method based on metric space in a distributed environment.

Technical solution: The parallel top-k dominating query method based on metric space in a distributed environment according to the present invention comprises the following steps in order:

(1) Given the query input data object set Q and the distance formula d() in the metric space, the distance formula d() is used to measure the distance between the entire data object O and the query input data object set Q;

(2) According to step (1), a parallel calculation based on set ANN and k-skyband is proposed. The specific content of the parallel algorithm is:

(21) Use ANN(Q,k) for pruning:

According to the distance measurement function d() and the query input Q, calculate the distance Deal_Data_RDD between all data objects and the query input object and save it in each partition, and then solve the middle ANN(Q,k) of the partition separately in parallel for each partition, Finally, the ANN(Q,k) results of each partition are filtered through the reduce interface to obtain the global ANN(Q,k); the acquired global ANN(Q,k) is broadcast to each node, and the ANN(Q,k ) to filter the original data set, and finally get the candidate set KANN(Q,k)_RDD, KANN(Q,k)_RDD must contain the final top-k dominant result set D, the filtering rule is not to be used by ANN(Q,k)_RDD K) is dominated by the object;

(22) Use k-skyband pruning:

Since the obtained KANN(Q,k)_RDD may be very large, it would be very time-consuming to directly calculate the dominance scores of all objects in KANN(Q,k)_RDD, so use the k-skyband idea to find KANN(Q,k)_RDD The k-skyband in k)_RDD is further pruned to obtain the final candidate set GlobalCandidate(k-skyband);

(23) Obtain top-k dominance:

Calculate the dominance scores of all objects in GlobalCandidate(k-skyband), and then find the top-k ones with the highest domination scores, and return them as the top-k domination results.

Further, in the step (21), since the ANN(Q,k) of each partition is not necessarily the global ANN(Q,k), it is necessary to compare the distances of the ANN(Q,k) of each partition one by one The far and near finally get the global ANN(Q, k).

Further, the detailed content of described step (23) is: carry out Cartesian product operation with the candidate set obtained in step (22) and original data set, then use the API interface of the ReduceByKey that Spark provides, obtain each candidate set dominate the score.

Beneficial effects: the present invention provides top-k dominance query based on metric space in a distributed environment, and proposes three distributed algorithms to solve top-k domination, by making full use of parallel computing between nodes in a distributed environment Its features, through pruning and sorting, greatly improve the performance of top-k dominated query based on metric space in a large data set environment, speed up query, and provide services for users' decision-making; specifically, it includes the following advantages:

(1) A parallel calculation skyline method is proposed, so that each partition can solve the skyline at the same time, so that the skyline can be solved quickly to obtain the top-k dominant result set;

(2) A method of parallel computing k-skyband is proposed, and each partition solves k-skyband independently without affecting each other, and the result can be obtained without looping by using the characteristics of k-skyband;

(3) It is proposed to use the collective ANN pruning first, and then calculate the k-skyband method in parallel. Effective pruning reduces the comparison operations between data, thereby speeding up the query.

Description of drawings

The flowchart of DAKDA algorithm in Fig. 1 the present invention;

Fig. 2 is a schematic diagram of the impact of the size of k on the query in the embodiment;

Fig. 3 is a schematic diagram of the influence of the size of m on the query in the embodiment;

Fig. 4 is a schematic diagram of the influence of the size query of c in the embodiment;

Fig. 5 is the extensibility contrast chart of each algorithm among the present invention;

Fig. 6 is a distributed processing diagram of the present invention;

Fig. 7 is an example diagram of the present invention.

detailed description

The technical solutions of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the embodiments.

The definitions of the symbols and parameters involved in the following are shown in Table 1:

Table 1 Symbol Description

Definition 1 (KNN(q,k)): Given a data set O, d() is a measure function, and o∈O, the k-nearest neighbor of object o is KNN(o,k), KNN(o,k) Indicates the k closest objects to object o.

Definition 2 (ANN(Q,k)): Given a data set O, d() is a measurement function, Q represents a set of query input objects Q={q ₁ ,q ₂ ,…,q _m }, ANN( Q,k) represents the k closest objects to Q. Choosing a reasonable set distance function d() will affect the query. Generally speaking, set distance functions include: minimum, maximum, average, etc.

Definition 3 (Domination in Metric Space): If (O,d()) is a metric space, Q represents a set of query input objects Q={q ₁ ,q ₂ ,...,q _m }. Then for an object o∈O, the set of distances between it and all objects in Q is:

adist(o,Q)={d(o,q ₁ ),d(o,q ₂ ),…,d(o,q _m )}

When the object p∈O, if o<p, then:

This dominance is measured by the size of the distance.

Definition 4 (metric-based top-k domination): Given a set of query inputs Q, and a distance metric function d(). According to the dominance relationship in the metric space, if the data object o _i ∈ O, the dominance score of the object o _i is:

dscore=|{p∈O|o<p}|, where Return the k objects with the highest dominance score, which is the top-k dominance query result set based on the metric space.

As shown in Figure 7, the top-k dominant query based on the metric space in this embodiment first queries the input Q={q ₁ , q ₂ }, the distance metric function d() used is the Euclidean distance, top-k The domination result of 1 is o ₁ , because the distances from o ₁ to q ₁ and q ₂ are smaller than all points outside the circle (including on the circle), only the object o ₂ is not dominated by o ₁ (because the distance from o ₂ to q ₁ is less than o ₁ distance to q ₁ ), if there are n data objects in the space, the dominance score of o ₁ is dscore(o ₁ )=n-1, and o ₂ does not dominate objects o ₁ and o ₃ at least, so the dominance score of o ₂ dscore (o ₂ )≤n-2, then dscore(o ₁ )>dscore(o ₂ ), so top-1 is dominated by o ₁ .

Define 5 (k-skyband) the entire data space, At most k-1 objects dominate the object o, and the set of this series of o is k-skyband.

Theorem 1: top-k dominates the result set

Proof. Counter-evidence, assuming there is an object o ₁ ∈ D, and the number of objects dominating o ₁ >k-1, so there must be k objects whose dominance score dscore≥o.dscore+1, at this time Contradiction, so top-k dominates the result set Proven.

Theorem 2: Query input set Q, k objects {o ₁ ,o ₂ ,…,o _k }∈O of ANN(Q,k), by (in Indicates not dominated) to form a set KANN(Q,k), where kANN(Q,k) contains the object ANN(Q,k) itself, and top-k dominates the result set

Proof. Let the 1-neighbor object of ANN(Q,1) query object Q be o, because all objects in D-1ANN(Q,1) are dominated by object o, so the top-1 dominance must be in 1ANN(Q ,1). If the top-1 dominance is not object o, it can be seen from the above that the object with the second highest dominance score must be in the set 1ANN(Q,1); if the top-1 domination is object o, then it can be seen from the above that the object with the second highest domination The object must be in the set 2ANN(Q,2), and so on, we know that top-k dominates the result set Proven.

All the following algorithms are implemented on the spark platform::

(1) Skyline-based top-k dominance algorithm (DSDA)

In the existing DSDA, firstly, the data set is randomly assigned to each node, and then the mappartition interface in spark is used to implement the calculation skyline algorithm in the Mappartition interface, so that the skyline of each partition can be obtained, and finally the skyline of each partition can be obtained The pairwise comparison obtains the global skyline, and the object with the highest dominance score in the returned skyline is the result set dominated by top-k. The final result set can be obtained by performing k loops sequentially.

(2) top-k dominance algorithm (DKDA) based on k-skyband

The existing DKDA parallelizes the algorithm in the spark cluster, and the idea of the parallel algorithm is similar to skyline. According to Theorem 1, it can be known that the top-k dominating result set So first find the k-skyband, and then return the k objects with the highest dominance score from the k-skyband as the top-k dominating result set.

First, the data set is randomly assigned to each node, and then the Mappartition interface in spark is used to implement the calculation k-skyband algorithm in the Mappartition interface, so that the k-skyband of each partition can be obtained, and finally the k-skyband of each partition The pairwise comparison obtains the global k-skyband, and the object with the highest dominance score in the returned k-skyband is the result set dominated by top-k. Compared with the skyline method, the advantage of this method is that it does not need to perform k cycles, but it is very time-consuming to solve the k-skyband of the original data set. (3) Parallel top-k dominance algorithm (DAKDA) based on collective ANN pruning and k-skyband

Because Algorithm 1 needs to perform k cycles, the query time increases with the increase of k, and Algorithm 2 is very time-consuming to solve the original data set k-skyband, so the present invention can perform pruning.

In the present invention, according to Theorem 1, it can be seen that top-k dominates the result set According to Theorem 2, we know that top-k dominates the result set Since solving k-skyband is more time-consuming than solving KANN(Q,k), first use the set ANN to remove the data that is not a candidate set to obtain the candidate set KANN(Q,k), and then solve k-skyband in KANN(Q,k) , and finally return the k results with the highest domination scores from the k-skyband as top-k dominance. The steps are shown in Figure 1:

Step 1: Pruning with ANN(Q,k)

As shown in Phase 1 of Figure 1 below, the data needs to be processed according to the distance measurement function d() and the query input Q to obtain the distance between each object and the query object. Deal_Data_RDD is stored in each partition, and then the median ANN( Q,k), and finally get the global ANN(Q,k). Use the global ANN(Q,k) to filter the original data set to get the candidate set KANN(Q,k)_RDD. According to Theorem 2, it can be known that KANN(Q,k)_RDD must contain the final top-k dominant result set D .

Step 2: Use k-skyband pruning

As shown in the second stage of Figure 1 below, since the obtained KANN(Q,k)_RDD may be very large, it is also very time-consuming to directly calculate the dominance scores of all objects in KANN(Q,k)_RDD, so use k- The idea of skyband is to find the k-skyband in KANN(Q,k)_RDD and further pruning to get the final candidate set GlobalCandidate(k-skyband). According to Theorem 1, it can be known that GlobalCandidate(k-skyband) must contain the final top-k dominant result set D.

Step 3: Get the top-k dominant result set

As shown in Phase 3 of Figure 1 below, Cartesian product operation is performed on the candidate set and the original data set to form <key, value>, where the key represents the candidate set, and if the candidate set dominates a piece of data in the original data set, the value is 1. Otherwise, it is 0; finally, the dominance scores of all objects in the GlobalCandidate (k-skyband) are obtained through the API interface of ReduceByKey, and then the top-k ones with the highest dominance scores are found.

Example 1:

This embodiment is completed on a 7-node spark distributed cluster. Spark is built on hadoop, using hadoop's yarn resource manager and HDFS file storage system. Among the 7 nodes, the master node is both a driver node and a worker node, and the remaining 6 nodes are all worker nodes. All algorithms are written in Scala language, and the basic configuration is shown in Table 2:

Table 2 Experimental environment configuration

As shown in Figure 2 to Figure 5, the experimental part mainly evaluates the three algorithms of DSDA, DKDA, and DAKDA from the following aspects: the impact of the number of partitions num on the query time (select a reasonable number of partitions), and the impact of the returned result k on the query , the impact of the size of the query input set Q on the query time, the comparison of the candidate sets of each algorithm, and the scalability of the algorithm. The default settings of the parameters in the experiment are shown in Table 3 below, where the coverage rate c = the radius of the smallest circle covering the input Q / Minimum circle radius to cover all datasets.

Table 3 Experiment default parameter configuration

First analyze the real large data set: ZILLOW data set, the original data set has 2,245,109 records, because some records have missing attribute values, the size of the deleted data set is 1,771,107, and there are 5 attributes in total. The distance formula for metric spaces uses the Manehattan distance. The specific process is shown in Figure 1. As shown in Figure 6, the data set is evenly divided into each slaver node, and then each node independently executes the algorithm proposed above to obtain a candidate set, and finally aggregates to obtain a top-k dominance result set.

Given m=5, Experiment 1 evaluates the performance of each algorithm as the number k of returned results changes. As shown in Figure 2, it is found that the DSDA algorithm changes significantly with k, while the DAKDA algorithm changes slightly with k, indicating that the DSDA algorithm is more sensitive to k.

Given k=10, Experiment 2 evaluates the performance of each algorithm as the size m of the query set Q changes. From Figure 3, we found that with the increase of m, the algorithm DKDA increases sharply.

Given k=10, m=5, Experiment 3 evaluates the performance of each algorithm as the coverage c of the query set Q changes. As shown in Figure 4: DSDA algorithm query is the slowest when the coverage rate is large. The scalability of the method of the present invention is shown in FIG. 5 .

As can be seen from the above-mentioned embodiment 1, for a given data set, according to the user's query input and the distance formula in the given metric space, the present invention proposes a top-k dominant parallel scheme suitable for large data sets; The k-skyband result set contains the characteristics of the top-k dominant result set. First, the set k-nearest neighbor pruning is used to obtain the candidate set, and then the k-skyband of the candidate set is obtained, and finally the top-k dominant result is solved.

This method based on k-skyband and ensemble ANN is compared with the traditional method of using skyline to solve top-k dominance, and the method of simply using k-skyband to solve top-k domination. It screens the data, reduces the number of comparisons between data, and speeds up query speed. The present invention is implemented in parallel on the spark platform, because the top-k dominant query based on metric space is only a stand-alone algorithm at present, and what the present invention proposes is a parallel algorithm, which is much faster than a stand-alone and the result of embodiment 1 just proves this conclusion, Therefore, the present invention parallelizes the traditional skyline-based and k-skyband-based methods, the method query speed is faster, and it is applicable to larger input sets or massive data sets.

Claims (3)

1. under a distributed environment, top-k based on metric space arranges querying method, it is characterised in that: include following successively The step that order performs:

(1) given range formula d () inquired about in input data object set Q and metric space, range formula d () is used for weighing Measure the distance between whole data object O and inquiry input data object set Q；

(2) proposing to calculate parallel based on set ANN and k-skyband according to step (1), the particular content of this parallel algorithm is:

(21) utilize ANN (Q, k) beta pruning:

The distance between all data objects and inquiry input object is calculated according to distance metric function d () and inquiry input Q Deal_Data_RDD also saves it in each subregion, the middle ANN of the most each subregion this subregion of independent Parallel implementation (Q, K), finally by the ANN of each subregion (Q, k) result carry out screening by reduce interface obtain the overall situation ANN (Q, k)；To obtain (Q, k) is broadcast on each node the overall ANN taken, and (Q k) goes to filter original data set, finally obtains candidate to utilize ANN (Q, (Q necessarily comprises last top-k and arranges result set D collection KANN in k) _ RDD for k) _ RDD, KANN；Wherein, (Q k) is ANN Refer to query set Q k-NN, the rule of filtration be not by ANN (Q, k) in object arranged；

(22) utilize k-skyband beta pruning: utilize k-skyband thought, find KANN (Q, the k-skyband in k) _ RDD, so Rear beta pruning further obtains final Candidate Set GlobalCandidate (k-skyband)；

(23) acquisition top-k domination result set:

Calculate the domination mark of all objects in GlobalCandidate (k-skyband), then find out top-k Zhi Peifen Number is the highest, returns and arranges result as top-k.

Under distributed environment the most according to claim 1, top-k based on metric space arranges querying method, and its feature exists In: in described step (21), due to each subregion ANN (Q, k) be not necessarily the overall situation ANN (Q k), then needs each point The ANN in district (Q, distance k) comparing distance one by one finally give the ANN of the overall situation (Q, k).

Under distributed environment the most according to claim 1, top-k based on metric space arranges querying method, and its feature exists In: the detailed content of described step (23) is: the Candidate Set obtained in step (22) and raw data set are carried out cartesian product Computing, then uses the api interface of the ReduceByKey of Spark offer, obtains the domination mark of each Candidate Set.