CN106909639A - A kind of chain type Multi way spatial join Query Processing Algorithm based on Spark - Google Patents

Abstract

The invention discloses a Spark-based chained multi-path spatial connection query processing algorithm, which comprises the following steps: Step 1: Divide the entire data space into many grid units of the same size, and use Z-order filling curve technology for each grid units for encoding; step 2: project each spatial object in the m-way spatial connection dataset to the corresponding grid unit according to its position in the data space; step 3: if the condition i<m is satisfied, then Two data sets RDDresult _new , RDD _i perform spatial join operation Overlap; step 4: i=i+1, execute step 3 until the condition i<m is not satisfied; step 5: perform the last spatial join operation Overlap. The invention is a Spark-based chained multi-way spatial join query processing algorithm, which has significantly improved processing efficiency and reduced calculation cost.

Description

A Chained Multi-way Spatial Join Query Processing Algorithm Based on Spark

technical field

The invention relates to the technical field of spatial data query processing, in particular to a Spark-based chained multi-way spatial join query processing algorithm.

Background technique

Spatial join query is an important type of spatial data query, which widely exists in spatial data management. Spatial join query processing technology has always been a research hotspot in the field of spatial database management. Multi-way spatial join query is a commonly used spatial join operation, which retrieves all spatial objects that satisfy a certain spatial predicate (such as intersection, inclusion, etc.) from multiple spatial data sets, and is one of the most time-consuming spatial operations. First, its complexity and importance make it one of the important factors that determine the overall performance of the spatial data management system, so improving the efficiency of multi-way spatial join query processing has always been a research hotspot in the academic circles. Especially in recent years, with the rapid development and wide application of Internet of Things technology, earth observation technology and location-based service technology, the scale of spatial data has increased dramatically, and it has become an important type of big data. How to perform efficient multi-way spatial join query processing on this kind of spatial big data has become an important challenge in the field of spatial data management. The traditional spatial database-based processing technology has the problem of weak scalability, so it is difficult to meet the requirements of fast query and processing of spatial big data, and Spark, as a new type of distributed parallel processing platform for ultra-large-scale data, has been widely valued by people. It is also the key technology of big data processing. Therefore, combined with the large-scale data processing capabilities provided by the Spark distributed parallel processing platform, it has become an important means to solve the above challenges to study the efficient multi-way spatial join query processing method of spatial big data.

In the multi-way spatial join query processing, the existing methods mainly have the following problems: (1) The traditional multi-way join processing method based on the spatial database mainly adopts a centralized processing method, which has poor scalability and is difficult to meet the needs of large spaces. Requirements for fast query and processing of data; (2) Some existing popular algorithms, such as dynamic programming algorithms, hybrid join algorithms, etc., mostly build indexes in a centralized manner, and the efficiency is relatively low for massive data join queries; (3) Existing The distributed processing method is mainly based on the Hadoop platform, and focuses on the optimization of general multi-way connection query processing. There are problems such as excessive data replication and weak filtering capabilities, which affect the efficiency of query processing; (4) the latest distributed The multi-way spatial join algorithm is that Gupta et al. proposed two multi-way spatial join query processing algorithms based on MapReduce Controlled-Replicate and ε-Controlled-Replicate. Controlled-Replicate divides and replicates the spatial objects in various connection datasets to all grid cells in the fourth quadrant, and then performs multi-way connection operations. Apparently, this method has caused a large number of spatial objects to be copied, which affects the efficiency of connection processing. For this reason, the author proposes an improved multi-way spatial join query processing algorithm ε-Controlled-Replicate, which reduces data replication to a certain extent and improves query processing efficiency, but there is still the problem of excessive data replication. For the Spark platform, too much data replication will cause too much data to be loaded into the memory at one time, which cannot make full use of the advantages of Spark's memory-based computing, and will also lead to low query efficiency and other problems.

After in-depth research on the above problems and corresponding solutions, they can be applied to related application fields such as connection query processing of spatial big data. For this reason, the present invention proposes a kind of spatial multi-way connection query processing algorithm under the Spark platform, this algorithm is mainly aimed at chain type multi-way spatial connection query, adopts grid-based data space division method, and combines Z-order coding to Realize data division and coding, and perform data projection and replication according to the spatial location of data objects. During the connection process, the algorithm adopts the boundary filtering method to reduce useless connection data, thereby reducing the redundant calculation of subsequent connections, as well as the redundant projection and duplication of connected objects. And use the duplicate avoidance strategy to reduce the output of duplicate results, thereby reducing the cost of subsequent connection calculations and improving the efficiency of multi-way connection query processing.

Contents of the invention

In view of the defects existing in the prior art, the purpose of the present invention is to provide a Spark-based chained multi-way spatial join query processing algorithm, which mainly focuses on the chained multi-way spatial join query processing problem, and focuses on reducing the filtering stage The amount of spatial data replication and calculation can reduce the cost of subsequent connection calculations and improve query processing efficiency. The algorithm also has good adaptability and scalability.

In order to achieve the above object, technical scheme of the present invention:

A spark-based chained multi-way spatial join query processing algorithm, comprising the steps:

Step 1: Use the grid division method to divide the entire data space into many grid units of the same size, and use Z-order filling curve technology to encode each grid unit;

Step 2: Project each spatial object in the m(m>2) way spatial connection data set R ₁ , R ₂ ,..., R _m to the corresponding grid unit according to its position in the data space, and form a A series of key-value pairs, store the projection results in the elastic distributed data sets RDD ₁ , RDD ₂ , ..., RDD _m , set the loop variable i=2, and set the intermediate result data set RDDresult _new =RDD ₁ ;

Step 3: If the condition i<m is satisfied, perform the spatial join operation Overlap(RDDresult _new , RDD _i ) on the two datasets RDDresult _new and RDD _i . In the calculation process, operations such as data aggregation, boundary filtering, spatial connection calculation, repetition avoidance, and data replication are performed in sequence, and finally an intermediate result data set RDDresult _new is formed, that is, RDDresult _new =Overlap(RDDresult _new ,RDD _i );

Step 4: i=i+1, execute step 3 until the condition i<m is not satisfied;

Step 5: Execute the last spatial connection operation Overlap(RDDresult _new ,RDD _m ). During the calculation process, data aggregation, boundary filtering, and spatial connection calculations are performed in sequence, and the results are directly output to form the final spatial connection result set and saved to HDFS file system.

Further, the data division and encoding method is: using a grid-based division method to divide the entire data space into n grid units of equal size, using Z-order filling curves to encode the grid units, and the spatial data object According to its position, it is projected to each grid unit, and all grid units are mapped to multiple Executor execution units by means of Hash, so that the entire processing task is divided into multiple parallel processing tasks.

Further, the projection of the spatial object is: mapping the spatial data object to the corresponding grid unit according to its location, let C=(c ₁ ,c ₂ ,...,c _n ) represent a data space division, c _i Represents each grid unit; Let R be a set of spatial objects to be connected. If a spatial object _u∈R , its MBR overlaps with the grid unit ci, and the _ci is the Z of the grid unit -order encoding, the object _u is mapped to the grid unit ci, and the corresponding key-value pair (ci, _u ) is generated. If a spatial object overlaps with multiple grid units, multiple key-value pairs.

Further, step 3 specifically includes the following steps:

Step 3-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ;

Step 3-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation;

Step 3-3: Execute the duplicate avoidance strategy, form the intermediate result of the connection, and perform data copy operation on the intermediate result of the connection, and finally form a new intermediate connection result data set RDDresult _new .

Further, step 5 includes the following steps:

Step 5-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ;

Step 5-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation;

Step 5-3: Execute the duplicate avoidance strategy to form the final connection result data set RDDresult _new consisting of tuple pairs, and save it to the HDFS file system.

Further, the data copy operation is: for any tuple t in the intermediate connection result set T generated by the latest spatial join operation on the current grid unit, if ts is a spatial object related to the next spatial join operation , then if _ts overlaps with a certain grid unit ci, copy the tuple t to the grid unit _ci and generate the corresponding key-value pair ( _ci ,t).

Further, the filtering strategy is: in the process of executing the connection operation in parallel, adopt a corresponding filtering strategy to remove the tuples that cannot generate the connection result, and only copy the tuples that may generate the connection result.

Further, the filtering strategy includes two parts:

Boundary filtering, the boundary filtering is as follows: before performing the connection operation, first count the boundary MBR of the spatial object related to the subsequent spatial connection in the completed connection intermediate result, and use the MBR to filter out the subsequent data set to be connected. MBR disjoint spatial objects, thereby reducing the calculation cost of subsequent connections;

Filtering in the copy stage, the filter in the copy stage is: in the process of multi-way join query processing, it is necessary to perform data copy operations on the intermediate results after the first few joins are processed, and only copy them to other grids that may generate join results In the unit, so as to avoid the loss of connection results, in the copying of intermediate connection results, only the intermediate results containing cross-grid connection objects are copied.

Further, the repetition avoidance strategy is: when two spatial objects spanning multiple grid units are connected, only the grid unit where the intersection point of the lower left corner of the new object formed by the two overlapping is responsible for the output result.

Compared with the prior art, the beneficial effects of the present invention: the present invention is a Spark-based chained multi-way spatial join query processing algorithm, adopts the grid division method to divide the data space, and based on the location of the spatial object to For data projection and replication, the boundary filtering method is used in the calculation process to filter out useless connection objects, and by reducing the replication range and reducing data replication, the processing efficiency and calculation cost have been significantly improved, and it has a good adaptability and scalability.

Description of drawings

Fig. 1 is an example diagram of Z-order curve coding in the specific embodiment of the present invention;

Fig. 2 is a schematic diagram of dividing data and task mapping in a specific embodiment of the present invention;

Fig. 3 is an example diagram of performing projection and copying operations on data in a specific embodiment of the present invention;

Fig. 4 is an example diagram of boundary filtering described in the specific embodiment of the present invention;

Fig. 5 is an example diagram of the repetition avoidance described in the specific embodiment of the present invention;

6 is a schematic diagram of the processing flow of the Spark-based chained multi-way spatial join query processing algorithm in a specific embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.

A Spark-based chained multi-way spatial join query processing algorithm proposed by the present invention, the algorithm mainly focuses on the chained multi-way spatial join query processing problem, the key point is to reduce the spatial data copy and calculation amount in the filtering stage, thereby reducing the follow-up The connection calculation cost can be improved, and the query processing efficiency can be improved. The algorithm also has good adaptability and scalability.

Example 1:

A Spark-based chained multi-way spatial join query processing algorithm, the key technologies mainly include the following parts:

1) Data space division and coding: the whole space is divided into n equal-sized grid units by using the grid division method, and the grid units are encoded by using the Z-order filling curve, and the spatial data objects are projected to each grid unit according to its position. Grid units, and use the Hash method to map all grid units to multiple Executor execution units, so that the entire processing task is divided into multiple parallel computing tasks, thereby improving the execution performance of the entire algorithm in a parallel manner.

As shown in Figure 1, in order to maintain the spatial relationship between spatial objects, a space-filling curve is used to encode the grid unit. Figure 1 shows the Z-order curve when the number of bits is 1 or 2. A Z-order curve is a space-filling curve. Z-order (z-sorting) technology is to use bits to represent the attribute information of spatial objects, and then use the loop method to decompose the data space, and the divided subspace will get a set of numbers, which is called the z of the subspace -Sort value, and serve as the Key value of the subspace data object.

Figure 2 shows an example of task division mapping. It can be seen from the figure that each divided grid unit and the data set divided on it are allocated to n Executor execution units in the Spark platform through Hash mapping to parallelize implement.

2) Spatial object projection: it is to map the spatial data object to the corresponding grid unit according to its location. Let C=(c ₁ ,c ₂ ,…,c _n ) represent a data space division, and c _i represent each grid unit; let R be a set of spatial objects to be connected, if a spatial object u∈R , its MBR overlaps with the grid unit ci (ci is the Z-order code of the grid unit), then the object _u is mapped to the grid unit _ci , and the corresponding key-value pair ( _ci , _u ), if a spatial object overlaps with multiple grid cells, multiple key-value pairs will be formed accordingly. The projection operation can be expressed as:

3) Data replication: In multi-way connection query processing, multiple connection operations between multiple data sets are required, and data replication is to copy the intermediate results of the latest spatial connection on the current grid unit to other related network cell, so that the subsequent connection operation, the result is similar to the projection operation, which will generate a series of key-value pairs. If t ∈ T is the tuple in the intermediate result of the connection, and ts is the object to be connected in the subsequent space, then if _ts overlaps with a certain grid unit ci, then the tuple t is copied to the grid unit _ci , and generate the corresponding key-value pair (c _i ,t). A copy operation can be expressed as:

Figure 3 is an example of data projection and copy operations, from which it can be seen that spatial objects are projected to overlapping grid cells. Object r ₁ is projected to grid cells 6 and 12, r ₂ is projected to cells 9 and 12, r ₃ is projected to cells 9 and 11, that is, Project(r ₁ ,C)={(6, r ₁ ),(12,r ₁ )}, Project(r ₂ ,C)={(9,r ₂ ),(12,r ₂ )}, Project(r ₃ ,C)={(9,r _{2 )} ),(11,r ₂ )}. When executing r ₁ , r ₂ and r ₃ to perform multi-way connection sequentially, since r ₂ overlaps with grid unit 9, the intermediate result of the connection between r ₁ and r ₂ in grid unit 12 (r ₁ , r ₂ ) to be copied to grid unit 9 to form a key-value pair (9,(r ₁ ,r ₂ )), so as to realize the subsequent connection operation with the spatial object r ₃ in grid unit 9, avoiding the loss of the connection result .

4) Filtering strategy: In the process of executing the connection operation in parallel, the boundary filtering strategy is adopted to remove the tuples that cannot produce connection results, and only copy the tuples that may have results, which greatly reduces the cost of storage and subsequent calculations. Specifically, the following two filtering strategies are included:

A: Boundary filtering: first count the boundary MBR of the related connection objects in the connection results of the previous several times, and use this MBR to filter out the spatial objects that are not intersected with the MBR in the subsequent data set to be connected, thereby reducing the calculation cost of subsequent connections:

Figure 4 is an example of boundary filtering. In the figure, three data sets R, S and T are followed by a three-way connection operation The spatial object projected into grid cell 3 is shown in the figure, The results are (r ₁ , s ₁ ), (r ₁ , s ₂ ), (r ₁ , s ₃ ), and the objects in the corresponding S collection in the previous connection result set are s ₁ , s ₂ and s ₃ , its boundary MBR is shown by the dotted line in the figure. When performing connection operations with objects in the data set T, the spatial objects t ₁ , t ₄ and t ₅ , which avoids the connection operation of these spatial objects with s ₁ , s ₂ and s ₃ respectively, thereby greatly reducing the cost of subsequent calculations.

B: Copy stage filtering: In the process of multi-way connection query processing, it is necessary to perform data copy operations on the intermediate results after the first few connection processes, copy them to other grid units that may generate connection results, and perform subsequent connections operation to avoid losing connection results. In the copying of intermediate connection results, only intermediate results involving cross-grid connection objects are copied, thereby avoiding redundant copying.

5) Duplication avoidance strategy: When two spatial objects spanning multiple grid units are connected, only the grid unit where the intersection point of the lower left corner of the new object formed by the two overlaps is responsible for outputting the result, that is, Only one grid unit is responsible for outputting results, which avoids repeated output of results and reduces processing costs.

Figure 5 shows an example of duplicate avoidance, where the object s ₁ in the S set is projected to its overlapping grid cells 2, 3, 6, 8, 9, 12, and the object r ₁ in the R set is It is projected to grid units 3, 6, 9, and 12, and the _r2 object is projected to four grid units 8, 9, 10, and 11. If duplication avoidance is not performed, during connection processing, grid units 3, 6, 9, and 12 will output the same connection result (r ₁ , s ₁ ), and grid cells 8 and 9 will also output the same connection result (r ₂ , s ₁ ), obviously duplication occurs. According to the proposed repetition avoidance strategy, as shown in Figure 5, the grid unit where the lower left corner of the object formed by the overlapping part of the object (shown by points P and Q in the figure) is located is responsible for outputting the result, that is, the grid unit 3 is responsible for processing the connection result (r ₁ , s ₁ ) of output r ₁ and s ₁ , grid unit 8 is responsible for processing the connection result (r ₂ , s ₁ ) of output r ₂ and s ₁ , this strategy avoids repeated processing and The repeated output of the results reduces the cost of subsequent processing.

Chained multi-way spatial join query Q _m =Overlap(R ₁ ,R ₂ ,R ₃ ,...,R _m ), according to its definition, can be expressed as Q _m =Overlap(...Overlap(Overlap(R ₁ ,R ₂ ), R ₃ ),..., R _m ), the processing flow of the Spark-based chained multi-way spatial join query processing algorithm proposed by the present invention is shown in Figure 6, and mainly includes the following steps:

A: According to the grid partition coding method, project the multi-connection data set R ₁ , R ₂ , R ₃ ,...,R _m , and use the coded value as the Key value, and set the identity of each spatial object and its attributes such as MBR Information is used as the Value value to form a series of key-value pairs, and the projection results of the data sets R ₁ , R ₂ , R ₃ ,...,R _m are placed in the elastic distributed data sets RDD ₁ , RDD ₂ , RDD ₃ , ..., RDD _m ;

B: Calculate Overlap(R ₁ , R ₂ ), that is, perform the Cogroup operation on RDD ₁ and RDD ₂ , gather the data in RDD ₁ and RDD ₂ together according to the Key value to obtain RDD _new , and use the boundary filtering strategy to perform RDD _new Filter, remove data objects that cannot have results, and then perform actual spatial connection operations, implement repetition avoidance strategies, and form connection intermediate results; perform data copy operations on connection intermediate results to form intermediate result data sets RDDresult _new ;

C: Calculate the connection operation between RDDresult _new and RDD ₃ according to the same calculation method as step B, and obtain the latest R ₁ , R ₂ , R ₃ connection intermediate result RDDresult _new . Adopt the same calculation method, sequentially calculate the connection operation between RDDresult _new and RDD ₄ , and RDD ₅ , ..., and RDD _m-1 , and finally get the data set R ₁ , R ₂ , R ₃ , ..., R _m-1 The connection intermediate result data set RDDresult _new ;

D: RDDresult _new and RDD _m execute the Cogroup operation to generate a new RDD _new , on this basis, perform boundary filtering, connection operation processing, and directly output the results to form data sets R ₁ , R ₂ , R ₃ ,…,R The final spatial join data set RDDresult _new of _m , and save the result to the HDFS file system. Since it is the last spatial join operation, no copy operation is required.

Example 2:

A Spark-based chained multi-way spatial join query processing algorithm, including the following steps: Step 1: Divide the entire data space into many grid units of the same size, and use Z-order filling curve technology to process each grid unit Encoding; Step 2: Project each spatial object in the m(m>2) way spatially connected data set R ₁ , R ₂ ,..., R _m to the corresponding grid unit according to its position in the data space, And store the projection results in elastic distributed datasets RDD ₁ , RDD ₂ , ..., RDD _m . Set loop variable i=2, intermediate result data set RDDresult _new =RDD ₁ ; Step 3: If the condition i<m is satisfied, perform spatial join operation Overlap(RDDresult _new , RDD _i for two data sets RDDresult _new , RDD _i ), in the calculation process, operations such as data aggregation, boundary filtering, spatial connection calculation, repetition avoidance and data replication are performed in sequence, and finally a new intermediate result data set RDDresult _new is formed, and RDDresult _new = Overlap(RDDresult _new , RDD _i ); steps 4: i=i+1, execute step 3 until the condition i<m is not satisfied; step 5: execute the last spatial connection operation Overlap(RDDresult _new ,RDD _m ), during the calculation process, data aggregation, boundary filtering, Calculate the spatial join, and output the result directly to form the final spatial join result set, and save it to the HDFS file system. The invention is a Spark-based chained multi-way spatial join query processing algorithm, which has significantly improved processing efficiency and reduced calculation cost, and has good adaptability and expansibility.

Example 3:

A spark-based chained multi-way spatial join query processing algorithm, comprising the steps:

Step 1: Use the grid division method to divide the entire data space into many grid units of the same size, and use Z-order filling curve technology to encode each grid unit;

Step 2: Project each spatial object in the m(m>2) way spatial connection data set R ₁ , R ₂ ,..., R _m to the corresponding grid unit according to its position in the data space, and form a A series of key-value pairs, store the projection results in the elastic distributed data sets RDD ₁ , RDD ₂ , ..., RDD _m , set the loop variable i=2, and set the intermediate result data set RDDresult _new =RDD ₁ ;

Step 3: If the condition i<m is satisfied, perform the spatial join operation Overlap(RDDresult _new , RDD _i ) on the two datasets RDDresult _new and RDD _i . In the calculation process, operations such as data aggregation, boundary filtering, spatial connection calculation, repetition avoidance, and data replication are performed in sequence, and finally an intermediate result data set RDDresult _new is formed, that is, RDDresult _new =Overlap(RDDresult _new ,RDD _i );

Step 4: i=i+1, execute step 3 until the condition i<m is not satisfied;

Step 5: Execute the last spatial connection operation Overlap(RDDresult _new ,RDD _m ). During the calculation process, data aggregation, boundary filtering, and spatial connection calculations are performed in sequence, and the results are directly output to form the final spatial connection result set and saved to HDFS file system.

Further, the data division and encoding method is: using a grid-based division method to divide the entire data space into n grid units of equal size, using Z-order filling curves to encode the grid units, and the spatial data object According to its position, it is projected to each grid unit, and all grid units are mapped to multiple Executor execution units by means of Hash, so that the entire processing task is divided into multiple parallel processing tasks.

Further, the projection of the spatial object is: mapping the spatial data object to the corresponding grid unit according to its location, let C=(c ₁ ,c ₂ ,...,c _n ) represent a data space division, c _i Represents each grid unit; Let R be a set of spatial objects to be connected. If a spatial object _u∈R , its MBR overlaps with the grid unit ci, and the _ci is the Z of the grid unit -order encoding, the object _u is mapped to the grid unit ci, and the corresponding key-value pair (ci, _u ) is generated. If a spatial object overlaps with multiple grid units, multiple key-value pairs.

Further, step 3 specifically includes the following steps:

Step 3-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ;

Step 3-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation;

Step 3-3: Execute the duplicate avoidance strategy, form the intermediate result of the connection, and perform data copy operation on the intermediate result of the connection, and finally form a new intermediate connection result data set RDDresult _new .

Further, step 5 includes the following steps:

Step 5-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ;

Step 5-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation;

Step 5-3: Execute the duplicate avoidance strategy to form the final connection result data set RDDresult _new consisting of tuple pairs, and save it to the HDFS file system.

Further, the data copy operation is: for any tuple t in the intermediate connection result set T generated by the latest spatial join operation on the current grid unit, if ts is a spatial object related to the next spatial join operation , then if _ts overlaps with a certain grid unit ci, copy the tuple t to the grid unit _ci and generate the corresponding key-value pair ( _ci ,t).

Further, the filtering strategy is: in the process of executing the connection operation in parallel, adopt a corresponding filtering strategy to remove the tuples that cannot generate the connection result, and only copy the tuples that may generate the connection result.

Further, the filtering strategy includes two parts:

Boundary filtering, the boundary filtering is as follows: before performing the connection operation, first count the boundary MBR of the spatial object related to the subsequent spatial connection in the completed connection intermediate result, and use the MBR to filter out the subsequent data set to be connected. MBR disjoint spatial objects, thereby reducing the calculation cost of subsequent connections;

Filtering in the copy stage, the filter in the copy stage is: in the process of multi-way join query processing, it is necessary to perform data copy operations on the intermediate results after the first few joins are processed, and only copy them to other grids that may generate join results In the unit, so as to avoid the loss of connection results, in the copying of intermediate connection results, only the intermediate results containing cross-grid connection objects are copied.

Further, the repetition avoidance strategy is: when two spatial objects spanning multiple grid units are connected, only the grid unit where the intersection point of the lower left corner of the new object formed by the two overlapping is responsible for the output result.

Although the specific implementation manner of the present invention has been described above, those who are familiar with the field should be understood that these are only illustrations, and the present invention is a chained multi-way spatial join query processing algorithm based on Spark, so illustrations are only In order to illustrate the core ideas of filtering strategy, duplicate avoidance strategy, connection processing flow, etc. Larger-scale experiments can be carried out later, and related algorithms can be further improved to improve the effects of data projection, replication and filtering. At the same time, it is also possible to consider combining indexing technology to further improve the performance of the algorithm without departing from the principle and essence of the present invention. The scope of the invention is limited only by the appended claims.

Claims (9)

1. a chained multi-way spatial join query processing algorithm based on Spark, is characterized in that: comprise the steps: Step 1: Use the grid division method to divide the entire data space into many grid units of the same size, and use Z-order filling curve technology to encode each grid unit; Step 2: Project each spatial object in the m(m>2) way spatial connection data set R ₁ , R ₂ ,..., R _m to the corresponding grid unit according to its position in the data space, and form a A series of key-value pairs, store the projection results in the elastic distributed data sets RDD ₁ , RDD ₂ , ..., RDD _m , set the loop variable i=2, and set the intermediate result data set RDDresult _new =RDD ₁ ; Step 3: If the condition i<m is satisfied, perform the spatial join operation Overlap(RDDresult _new , RDD _i ) on the two datasets RDDresult _new and RDD _i . In the calculation process, operations such as data aggregation, boundary filtering, spatial connection calculation, repetition avoidance, and data replication are performed in sequence, and finally an intermediate result data set RDDresult _new is formed, that is, RDDresult _new =Overlap(RDDresult _new ,RDD _i ); Step 4: i=i+1, execute step 3 until the condition i<m is not satisfied; Step 5: Execute the last spatial connection operation Overlap(RDDresult _new ,RDD _m ). During the calculation process, data aggregation, boundary filtering, and spatial connection calculations are performed in sequence, and the results are directly output to form the final spatial connection result set and saved to HDFS file system. 2. the chained multi-way spatial join query processing algorithm based on Spark according to claim 1, is characterized in that: described data division and coding method are: adopt grid-based division method to divide whole data space into n For grid units of equal size, the Z-order filling curve is used to encode the grid units, and the spatial data objects are projected to each grid unit according to their positions, and all grid units are mapped to multiple Executor execution units by Hash method , so that the entire processing task is divided into multiple parallel processing tasks. 3. the chained multi-way spatial join query processing algorithm based on Spark according to claim 1, characterized in that: the spatial object projection is: the spatial data object is mapped into the corresponding grid unit according to its position, Let C=(c ₁ ,c ₂ ,…,c _n ) represent a data space division, and c _i represent each grid unit; let R be a set of spatial objects to be connected, if a spatial object u∈R , its MBR overlaps with the grid unit ci, and the _ci is the Z-order code of the grid unit, then the object _u is mapped to the grid unit _ci , and corresponding key-value pairs ( _ci , u), if a spatial object overlaps with multiple grid cells, multiple key-value pairs will be generated accordingly. 4. the chained multi-way spatial join query processing algorithm based on Spark according to claim 1, is characterized in that: step 3 specifically comprises the following steps: Step 3-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ; Step 3-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation; Step 3-3: Execute the duplicate avoidance strategy, form the intermediate result of the connection, and perform data copy operation on the intermediate result of the connection, and finally form a new intermediate connection result data set RDDresult _new . 5. the chained multi-way spatial join query processing algorithm based on Spark according to claim 1, is characterized in that: step 5 comprises the following steps: Step 5-1: Calculate Overlap(RDDresult _new , RDD _i ), that is, perform Cogroup operation on RDDresult _new and RDD _i according to the Key value, that is, gather the data in RDDresult _new and RDD _i together according to the Key value to obtain RDD _new ; Step 5-2: Use the filtering strategy to filter the RDD _new , remove the data pairs that cannot have results, and then perform the actual spatial connection operation; Step 5-3: Execute the duplicate avoidance strategy to form the final connection result data set RDDresult _new consisting of tuple pairs, and save it to the HDFS file system. 6. the chained multi-way spatial join query processing algorithm based on Spark according to claim 1, characterized in that: the data copy operation is: for the intermediate connection generated by the latest spatial join operation on the current grid unit For any tuple t in the result set T, if ts is a spatial object related to the next spatial join operation, if ts overlaps with a certain grid unit c _i , then copy the tuple t to the grid unit c _i , and generate the corresponding key-value pair (c _i ,t). 7. the chained multi-way spatial join query processing algorithm based on Spark according to claim 4, characterized in that: the filtering strategy is: in the process of performing the connection operation in parallel, adopt the corresponding filtering strategy to remove the impossible Tuples of concatenated results, and copy only those tuples that could have resulted in concatenated results. 8. the chained multi-way spatial join query processing algorithm based on Spark according to claim 7, is characterized in that: described filtering strategy comprises two parts: Boundary filtering, the boundary filtering is as follows: before performing the connection operation, first count the boundary MBR of the spatial object related to the subsequent spatial connection in the completed connection intermediate result, and use the MBR to filter out the subsequent data set to be connected. MBR disjoint spatial objects, thereby reducing the calculation cost of subsequent connections; Filtering in the copy stage, the filter in the copy stage is: in the process of multi-way join query processing, it is necessary to perform data copy operations on the intermediate results after the first few joins are processed, and only copy them to other grids that may generate join results In the unit, so as to avoid the loss of connection results, in the copying of intermediate connection results, only the intermediate results containing cross-grid connection objects are copied. 9. the chained multi-way spatial join query processing algorithm based on Spark according to claim 4, is characterized in that: described repetition avoidance strategy is: when two spatial objects across a plurality of grid units are connected, only Let the grid cell where the intersection point of the lower left corner of the new object formed by these two overlap is responsible for outputting the result.