KR20180077728A - Skyline querying method based on quadtree - Google Patents

Abstract

A skyline querying method based on a quadtree includes a step in which a computer device generates a point quadtree using a data set; a step in which the computer device determines a local skyline for each of a plurality of end nodes which exist in the quadtree; and a step in which the computer device determines a global skyline using the local skyline for the plurality of end nodes. Accordingly, the present invention can perform a very fast skyline query.

Description

[0001] SKYLINE QUERYING METHOD BASED ON QUADTREE [0002]

The techniques described below relate to skyline query techniques.

With the spread of the Internet and the popularization of smart devices, a lot of data is accumulating in the network. Big data services that provide information by analyzing recently accumulated data are attracting attention.

There are various techniques for data analysis. Among them, skyline queries are data analysis techniques that can help users and enterprises make decisions.

Korean Patent Publication No. 10-2015-0089542

There are a variety of technologies such as BNL, D & Q, SFS, Bitmap, NN, and BBS that use a single computer to compute the skyline. However, this technique has a disadvantage in that it requires a very long time or a very large memory space for a large amount of data because parallel processing using a plurality of computers is difficult. Meanwhile, there are algorithms such as MRBNL and PPPS that use parallel computer and multicore processor to compute the skyline in parallel, but these algorithms have a disadvantage that they can not make full use of parallel processing.

The technique described below is to provide a skyline query method that distributes data suitable for parallel processing using a point quadtree data structure.

A quad tree-based skyline query method includes the steps of a computer device generating a point quadtree using a data set, the computer device determining a local skyline for each of a plurality of end nodes present in the quadtree, And the computing device determining the global skyline using the local skyline for the plurality of end nodes.

The techniques described below enable a very fast skyline query by evenly distributing computation quantities to a plurality of computing devices in a parallel processing system such as MapReduce framework.

Figure 1 is an example of a flowchart for a quadtree-based skyline query method.
FIG. 2 is an example of a point quadtree and a local quadtree to which data is distributed.
Figure 3 is another example of a point quadtree in which data is distributed.
Figure 4 is another example of a point quadtree where data is distributed.
5 is an example in which a point quadtree is divided.
6 is a flowchart illustrating a process of generating a point quadtree.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include " should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms " comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The techniques described below relate to skyline query techniques. First, the skyline query is briefly described. Skyline queries are a technique for making decisions when there are multiple criteria.

Let p be the total set of tuples, and let p be the point with d dimension values. Let p be the p (1), p (2), ... , p (d) &gt;. (K) ≤ r (k) for all dimensions k and (ii) p (j) <r (k) for at least one dimension j when there are two tuples p and r contained in D j) defines that the tuple p dominates the tuple r. D's skyline is the set of all tuples that are not dominated by any other tuple, and the tuples included in the skyline are called skyline tuples.

Type of notebook price Weight (kg) A 40 2.0 B 60 2.5 C 70 1.0

For example, suppose that there is a notebook of the same kind as in Table 1, and a user wants to buy a "lightweight notebook at a low price." Notebook B is more expensive and heavier than notebook A, so the user will not consider the notebook B at all. On the other hand, the notebook C is more expensive than the notebook A, but since the weight is lighter, the user can not say that the notebook C is worse than A. That is, since p (j) <r (j) holds for the case where p (1) = 40 ≤ r (1) = 60, p (2) = 2.0 ≤ r The notebook A dominates the notebook B. Thus, the ruled notebook B is not included in the skyline as defined by the skyline. On the other hand, for a combination of other tuples A and C, B and C, one tuple does not dominate the other. Thus, the skyline of D is a set of A and C that have never been dominated.

The techniques described below can be performed in a distributed processing system. For example, the skyline query described below can be performed in a system based on the MapReduce framework, which is attracting attention in connection with big data processing.

Google MapReduce is a framework for efficiently supporting distributed computing. In the MapReduce framework, a large cluster is formed by connecting a large number of computers (nodes) via a network. Data is represented in the form of a key-value pair consisting of keys and values, and is divided and stored in a plurality of computers by a distributed file system. When performing the operation on the stored data, it is performed by a map function (function) and a reduce function (function), which are two functions to be set by the user. The map function takes one key-value pair stored in the distributed file system as input and prints out a few key-value pairs. All output key-value pairs list key-value pairs having the same key by the shuffling phase. The Reduce function takes a list of each distinct key value and a value that shares the key, outputs a few key-value pairs, and stores it in the distributed file system. One end of the MapReduce operation is done by ending all the Reduce functions, and you can perform other MapReduce operations using the output of the Reduce function.

Calling a map function using a key-value stored in a distributed file system, or executing a shuffling step to create a list of key-values with the same key value, does not require the user to intervene in the MapReduce framework It is processed automatically. Therefore, when the map function and the reduction function receive input, the user decides which operation is performed by each function to output a key-value pair, and the MapReduce framework executes the operation in a distributed manner. There are a variety of implementations of the MapReduce framework, but an open source folkship called Hadoop is widely used.

By using the MapReduce framework, it is easy to distribute a large number of computers using various computers. For example, to count the number of occurrences of a word in a file. Suppose that the name of a file is a key, and the contents of the file are stored in the distributed file system as key-value pairs. At this time, the user wants to calculate the kind of each word appearing in the file and the number of occurrences thereof. As described earlier, you only need to define map functions and reduce functions to suit your purposes. First, the file name - file content pair enters the input of the map function. At that time, the map function outputs the word contained in the file contents as a key, and sets the value to 1. Then, the key-value pairs of (word, 1) will be created and the number of key-value pairs in which the key is a specific word will be the number of occurrences of that particular word in the entire document. Therefore, the key-value pairs having the same key are gathered by the shuffling step, and are passed to the input of the reduction function. In the reduction function, the number of the key-value pairs can be known by counting the number of the key-value pairs. Thus, if we print out a key-value pair with the word as the key and the number of occurrences of the word as the value, we can know the number of occurrences of the word in all documents.

Key (file name) Value (file content) A Hello World B Hello Hadoop C Bye World

It is assumed that the three files and their contents are as shown in Table 2 above. Suppose that a key-value pair is represented in the form of (key, value). First, the three map functions are called with (A, "Hello World"), (B, "Hello Hadoop"), and (C, "Bye World" ("Hello", 1), ("World", 1) is output as a map function that receives input (A, "Hello World" Likewise ("Hello", 1), ("Hadoop", 1), ("Bye", 1), ("World", 1) will be output by the remaining two map functions. In the shuffling step, a total of six key-value pairs output by the map functions are grouped together with the same key. 1), (("Hadoop", 1)), (("World", 1) "Bye", 1)). Since there are a total of four different key values, four reduction functions are called with each key and key-value list input. First, the reduction function called by (("Hello", 1), ("Hello", 1)) outputs a total of two key-value pairs ("Hello", 2) It means that the word came out twice. Similarly, key-value pairs of ("World", 2), ("Hadoop", 1), ("Bye", 1) occur so we have "Hello" 2, "World" 2, "Hadoop" And "Bye" 1 times. In practice, this process is performed in parallel by using multiple computers by the MapleDesktop framework. Therefore, when a large amount of data is processed, the operation is performed faster than using one computer.

Now, a method of performing a skyline query using the same system as the MapReduce framework described above will be described. For convenience of explanation, the apparatus for performing the skyline query will be referred to as a computer apparatus. A computer device means a device capable of distributed processing.

Figure 1 is an example of a flowchart for a quadtree-based skyline query method 100. The computer device first receives data for analysis (110). Thereafter, the skyline query is performed in three stages. The computer device generates the quadtree using the input data (120). As will be described later, a quad tree means a point quad tree. The computer device determines 130 the local skyline among the tuples included in the end node of the quadtree. The computer device then determines a global skyline using local skylines, virtual maximum points, and sky filters (140). Each step will be described in detail below.

Create point quad tree

Data set D is a set of d dimensional vectors. You can create a point quad tree by sampling a portion of data set D. That is, the computer device can generate a point quad tree based on the sample set S belonging to D. If you use a local quadtree, you must set a certain threshold for quadtree branching. However, using a point quadtree does not require a threshold for node partitioning, and when dividing a node into child nodes, all the points belonging to one node are not skylines. FIG. 2 is an example of a point quadtree and a local quadtree to which data is distributed. FIG. 2 (a) is an example of a point quad tree, and FIG. 2 (b) is an example of a region quad tree. In the case of FIG. 2 (a), all the points in node 4 are dominated by point 1, so node 4 has no skyline point. On the other hand, when a local quad tree is used as shown in FIG. 2 (b), the point at node 4 is not removed because there is no point at node 1.

It is not a problem because you always divide the nodes of the region quad tree by the median value, but you have to decide which point to divide to use the point quad tree. You also have to decide when to stop the quadtree break.

First, it is possible to decide which point to divide into as follows. As shown in FIG. 2 (a), let us consider a case where data is divided into a point 1. A simple skyline operation algorithm determines if a point on node 1 dominates another point on node 1, a point on node 1 dominates another point on node 2, ... , And whether the point at node 4 dominates the other point in node 4 is 4 * 4 = 16 cases. And denote each (node x, node y). In this case (node 2, node 2) it means that each point of node 2 is dominated by the other points of node 2. Since there are four points in node 2, each point must be judged by three points excluding itself. That is, 4 * 3 comparisons are required. (Node 2, node 3), the node 4 has 4 points and the node 3 has 1 point. In summary (Node 1, Node 1), ... , (Node 4, node 4). In each case, the number of comparisons is determined by the number of points included in the node.

However, when the point quad tree of FIG. 2 (a) is used, it can be seen that the points at the node 3 do not dominate the points at the node 2. Therefore, when dividing by the point 1, the skyline can be obtained by comparing only the points on the node 3 and the points on the node 2. Since node 4 has already been dominated, it is not necessary to compare whether the point of node 2 dominates the point of node 4, and node 1 contains only point 1, so that the point on node 1 is not compared with the other points on node 1 do. In summary, you only need to find the number of skylines internally in 4 * 4 nodes (node 3, node 3), (node 2, node 2). Since there is only one point in node 3 (node 3, node 3), it is only necessary to compare 1 * 0 = 0 times and there are 4 points in node 2 (node 2 and node 2) Should compare 12 times.

Figure 3 is another example of a point quadtree in which data is distributed. As shown in FIG. 3, when the node is divided into two points (node 3 and node 3), 2 * 1 = 2 times and (node 2, node 2) Can be found. Thus, Figure 3 is a more efficient point-quad tree than Figure 2 (a).

For two-dimensional data, there are always two cases, but for multidimensional cases, there are more cases. In order to deal with the multidimensional case in general, the following Equation 1 is used. Equation 1 computes the number of comparisons that are reduced when a given node n is divided by a point p in the node. Therefore, dividing by the maximum number of comparisons is the most efficient point-quad tree.

는 자식 노드 n_c에 들어 있는 샘플 점들의 스카이라인 점들의 좌표의 최대 값으로 만든 점이다.

를 가상최대점이라고 한다. 예를 들어 점들이 (2,8), (4,5), (1,7)이 있다면 가상 최대점은 (max(2,4,1), max(8,5,7))=(4,8) 이다. 만약 점p가 이 가상 최대점을 지배하지 못하는 경우, 점 p는 자식 노드 n_c안에 포함된 어떠한 점도 지배하지 못한다. 수학식 1의 첫 번째 항은 지배되지 않은 노드 (n_r 이 아닌 모든 노드)의 점들이 비교하지 않아도 되는 점들의 횟수를 구한다. 수학식 1의 두 번째 항은 n_r에 있는 점들은 항상 스카이라인 점이 아니므로 줄어드는 비교 횟수를 의미한다.In Equation (1), S (n) is a set of sample points included in the region of the node n. Also n.child is the node being reliably controlled by a set of child nodes, n _r is the time point p divided the node n to a point p (e.g., nodes of FIG. 2 (a) 4). There is only one node that is reliably governed by point p in any dimension data.

Is the maximum value of the coordinates of the skyline points of the sample points contained in the child node n _c .

Is called a virtual maximum point. For example, if the points are (2,8), (4,5), (1,7), the virtual maximum point is max (2,4,1), max (8,5,7) , 8). If the point p does not dominate this virtual maxima, the point p does not dominate any points contained in the child node n _c . The first term of Equation (1) finds the number of points at which the points of an unguided node (all nodes other than n _r ) need not be compared. The second term in Equation (1) means the number of comparisons reduced because the points in n _r are not always skyline points.

If node n is divided, the number of comparisons to be reduced for all points in node n is calculated using Equation 1, and the quad tree is divided based on the minimum number of comparisons. At this time, it is not necessary to divide all the points in the node. Figure 4 is another example of a point quadtree where data is distributed. As shown in FIG. 4, point 3 is dominated by point 1, but if we divide the data by point 3, we must always make more comparisons than when we divide point 1. Therefore, it is desirable to select a skyline point in the sample, obtain the number of times of comparison when the skyline point is divided by that point, and divide the node by the point of greatest number of times.

5 is an example in which a point quadtree is divided. The sample set S = {p ₂ , p ₄ , p ₆ , p ₈ , p ₁₀ , p ₁₂ , p ₁₄ , p ₁₆ } belonging to the data set D is. FIG. 5 (a) shows a state in which points belonging to the sample set S are arranged at n ₀ , which is a root node. SL (S) = {p ₂ , p ₄ , p ₈ , p ₁₀ , p ₁₂ , p ₁₄ }. SL (P) denotes the skyline for the set P of points. Therefore, SL (S) means the skyline for the prime sample set S of FIG. 5 (a). The points belonging to the skyline become candidate candidates for dividing the node thereafter. That is, let C (n) be a candidate set for dividing node n, C (n ₀ ) = SL (S).

은 = {5,9}이다. 도 5(b)에서 가상 최대점을 별표로 표시하였다. p₄는

를 지배하지 못하기 때문에 S(n2)에 속한 점들 중 어느 점이 지역 스카이라인지 따질 때 p₄를 판단할 필요가 없다. Even if n ₀ is divided based on the points p ₄ as shown in 5 (b), S is _{S (n 1) = {p} 4}, S (n 2) = {p 2, p 10, p 14, p 16 }, S (n ₃ ) = {p ₈ , p ₁₂ }, and S (n ₄ ) = {p ₆ }. Since C (n ₂ ) = {p ₂ , p ₁₀ , p ₁₄ }, the virtual maximum point of n ₂

Is = {5,9}. In FIG. 5 (b), the virtual maximum points are marked with an asterisk. p ₄ is

It is not necessary to determine p ₄ when a point in S (n2) belongs to the local skyline.

Node n ₄ is pruned because the minimum point is p ₄ . p ₄ corresponds to a sky filter. Since p ₆ is located at the node n ₄ to be pruned, it is not necessary to determine if it dominates p ₆ for the remaining sample points except p ₆ . In Fig. 5 (b), the excluded nodes are indicated by deletion lines. (N ₀ , p _i ) in Equation (1) with respect to (n ₀ , p ₄ ) for all points belonging to C (n ₀ ). Therefore, we choose p ₄ as the point for dividing n ₀ .

Node n ₂ has more than n ₁ or n ₃ sample points. Figure 5 (c) illustrates the example split for n _2. Among candidate points belonging to the C (n _{2) p. 14,} because the size is the result of the equation (1) selects the p ₁₄ as a reference point for n _2.

FIG. 5C shows an example of dividing n ₂ . Dividing the n ₂ to n ₂ divide the assumption is the time to calculate a local and global skyline skyline reduced. In other words, it is necessary to find a reference point for dividing n _2, and confirm that dividing the node before the actual division reduces the skyline operation time. To determine this, we estimate the time required to find the skyline of each node. The method of predicting the skyline operation time may utilize a conventionally studied technique.

The number of skylines can be estimated as Alog ^B | N |. Where A and B are constants and N is the number of points in the set. To create a point quad tree, divide the sampled points into two groups S ₁ and S ₂ . The size of the group is not equal. Each divided group calculates its own skyline. Because it is a sample, it can be calculated quickly by using a simple algorithm. The constants A and B can be calculated by the following equation (2).

SL (S ₁ ) is a set of skyline points of group S ₁ , and SL (S ₂ ) is a set of skyline points of group S ₂ .

Now we can get the expected number of skylines of the points at each node of the point-quad tree. The computer device predicts the expected local skyline and global skyline computing time when using a given quadtree and splits the node if it is expected that further partitioning of the node in the currently configured quadtree will reduce the computation time.

으로 계산한다. 여기서 |S|는 쿼드트리를 만들기 위해 사용한 샘플 점의 수이고, |D|는 전체 데이터의 수이다. 전역 스카이라인 계산 시간은

으로 계산한다. |up(n)|은 노드 n에 있는 점들을 지배할 수도 있는 점들의 집합의 크기이다. Given node n in the quadtree, the local skyline computation time is

. Where | S | is the number of sample points used to create the quadtree, and | D | is the total number of data. The global skyline calculation time is

. | up (n) | is the size of the set of points that may dominate the points at node n.

을 예측할 수 있다.On the other hand, computer devices distribute certain tasks to individual machines capable of parallel processing. Therefore, the skyline operation time is expected based on the assumption that the skyline operation on the node is uniformly distributed to individual machines. In addition, the actual skyline operation is performed in accordance with the distribution of the skyline calculation time estimate. In consideration of the parallel processing system, the total skyline calculation time obtained by adding the local skyline operation time and the global skyline operation time as shown in Equation 3 below

Can be predicted.

Where m is the number of individual machines available in the distributed processing system.

The point quad tree generation process described above is summarized. FIG. 6 is a flowchart of a process 200 of generating a point quadtree. The computer device receives a sample point belonging to the node (210). Initially, it starts with the root node. Then, the computer device selects a reference point for dividing the corresponding node using Equation (1) as described above (220). The computer device estimates the skyline operation time when it is divided using the reference point (230). The computer device compares the operation time of the skyline using the node divided using the reference point with the operation time using the quad tree before division (240). If the computation time is reduced in the case of partitioning, the computer device divides the node using the selected reference point (250). If you can not split the node anymore, the quad tree is complete. The skyline is then determined based on the completed quad tree. The process of determining the skyline may be similar to the conventional technique.

Regional Skyline Decision

The local skyline is performed in a parallel processing system such as the MapReduce framework. It is assumed that the task is distributed to individual machines while predicting the computation time in the quad tree generation process described above. In the actual regional skyline determination, each individual machine performs computation on each node as it is distributed in the forecasting process.

The point quad tree determines the local skyline at the end node. The skyline is not determined for the nodes excluded from the quad tree generation process. As described above, an operation is performed on the samples included in the node to determine the skyline. Once the skyline is determined, the virtual maximum point and the sky filter are determined to reduce the number of times the dominant relationship is confirmed in the process. The virtual maximum point means the point having the largest value among the points constituting the skyline. The sky filter means the point at which the node has the most value. As described above, the virtual maximum point and the sky filter are used in the process of generating a quadtree.

Determining the global skyline

At the non-excluded quadtree end node, we decided to construct the local skyline, respectively. If the point p constituting the skyline is not dominated by the point p 'that dominates the virtual maximum point among the points constituting the local skyline, the point p is determined as a point constituting the global skyline.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

Claims (7)

The computer device generating a point quadrature using the data set;
Determining a local skyline for each of a plurality of end nodes present in the quadtree; And
Wherein the computer device determines a global skyline using a local skyline for the plurality of end nodes. The method according to claim 1,
Wherein the computer device generates the point quadtree by dividing a child node into the specific point where the number of comparisons is minimized when dividing a child node using a specific point. The method according to claim 1,
The computer device generates the point quadtree by partitioning a node having the data set into one point belonging to the node using the following equation,
Wherein the one point is a point among the points belonging to the subnodes that are not dominated by any one of the points for the skyline operation of the subnodes when the node is divided into a plurality of subnodes among the points belonging to the node Wherein the sum of the number of points and the number of points belonging to a sub node dominated by any one of the points is a maximum. The method according to claim 1,
The generating step
The computer device comprising: determining the specific point at which the number of comparisons to be reduced is maximum when dividing a node using a specific point; And
When the computer device divides a child node into the specific point, sampling points that are part of the points belonging to the divided child node into two sample sets, calculating a skyline for points belonging to the sample set, And dividing the child node using the specific point if the skyline operation time is reduced when dividing the child node into the specific point than the structure of the quad tree. The method according to claim 1,
Wherein the computer device determines the global skyline using a local skyline, a virtual maximum point, and a sky filter for the plurality of end nodes. The method according to claim 1,
Wherein the computer device is a system based on a MapReduce framework, the quad tree based skyline query method. The method according to claim 6,
Wherein a plurality of individual machines belonging to the MapReduce framework determine the local skyline or the power skyline in parallel,
Based on a quadtree that distributes the local skyline or power skyline determination operations to the respective machines such that the total computation time is minimized on an individual machine belonging to the MapReduce framework based on the number of points belonging to the end node Skyline query method.

Images