KR102036314B1 - Spatial web object retrieval system based on distributed processing and spatial web object retrieval method in using same - Google Patents

Abstract

The present invention relates to a distributed processing based spatial web object search system. The present invention provides an in-memory distributed processing unit for changing data stored in an external storage device into in-memory data used on a memory; A spatial web object generating unit generating spatial web objects by including spatial information and word information in the in-memory data changed by the in-memory distribution processor; And a spatial web object query unit which searches a spatial web object including corresponding spatial information and word information by querying the spatial web object generated by the spatial web object generator for specific spatial information and word information. It is characterized by.

Description

Distributed Web-based object search system and distributed Web-based object search method using this method {SPATIAL WEB OBJECT RETRIEVAL SYSTEM BASED ON DISTRIBUTED PROCESSING AND SPATIAL WEB OBJECT RETRIEVAL METHOD IN USING SAME}

Embodiments of the present invention relate to distributed processing based spatial web object retrieval systems and methods.

A spatial web object is a document that contains information that can identify information about a region, such as GPS coordinates, region name, and postal code. Recently, due to the development of mobile devices such as smart phones, such spatial web objects are created exponentially, and the usage of services for searching spatial web objects is increasing.

Since the spatial web object has both the location information and the word information of the document, to search for the spatial web object, the first search is performed using the word information or the location information, and the location information or the word information is reused in the search results. You need to do a second search to find the exact result.

Therefore, a conventional spatial search technique for searching only a space, a word search technique for searching only words, or a word index for information about a word is added to each node of a spatial index, or a spatial information for each node of a word index is searched for. Hybrid index type search technique that adds spatial index is not an efficient way to search spatial web objects.

Accordingly, the present inventors have invented a system that can show excellent performance in the search of spatial web objects by applying the existing spatial web object retrieval method to a distributed processing environment based to be suitable for analyzing a large amount of data.

Republic of Korea Patent Publication No. 10-2009-0072619 (2009.07.02.)

The present invention provides a distributed processing-based spatial web object retrieval system and a distributed processing-based spatial web object retrieval method that can efficiently use a distributed processing system by using spatial partitioning and improve the efficiency of searching for the divided space. Its purpose is to.

In addition, the present invention is a distributed processing-based spatial web object retrieval system that can show an excellent performance in the search of spatial web objects by applying a QP-tree that builds an index using both spatial information and word information for data retrieval And a distributed processing-based spatial web object retrieval method using the same.

In accordance with another aspect of the present invention, there is provided a distributed processing-based spatial web object searching system, including: an in-memory distributed processing unit for changing data stored in an external storage device into in-memory data used on a memory; A spatial web object generating unit generating spatial web objects by including spatial information and word information in the in-memory data changed by the in-memory distribution processor; And a spatial web object query unit which searches a spatial web object including corresponding spatial information and word information by querying the spatial web object generated by the spatial web object generator for specific spatial information and word information. It is characterized by.

The in-memory distribution processor may change data stored in the external storage device into an RDD which is an elastic distributed data set that is the in-memory data.

The spatial web object generating unit may include spatial information and word information, respectively, in the text data generated by the in-memory distribution processor to generate spatial web aldi, which is a set of spatial web objects. ) module; A Gridded RD module for generating a grid RD by dividing the spatial web Aldi using a space dividing algorithm and constructing each divided space into one partition; And an IndexedRDD module for generating an Indexeddid for building an index for searching for a spatial web object included in each partition of the Griddidi.

In addition, the GridDRD module may perform spatial partitioning on the spatial web Aldi using a STR algorithm or a BSP algorithm, which is a spatial partitioning algorithm.

In addition, the IndexedRDD module may use a QP-tree as an index of the spatial web object.

The spatial web object query unit may include: a Boolean Range Query module that receives the spatial information and the word information and searches for a document including all of the corresponding words among the spatial web objects within the corresponding spatial range; And a Boolean Top-k Query module that receives the spatial information and the word information and searches the K spatial web objects closest to each other containing the corresponding words at the corresponding spatial location.

In addition, the distributed processing-based spatial web object search method according to the present invention includes an in-memory distribution processing step of the in-memory distribution processing unit to change the data stored in the external storage to the in-memory data used on the memory; A spatial web object generation step of generating a spatial web object by including spatial information and word information in the in-memory data changed by the in-memory distributed processing unit; And a spatial web object querying step, wherein the spatial web object querying unit searches a spatial web object generated by the spatial web object generating unit for specific spatial information and word information and searches for a spatial web object including the spatial information and word information. It can include;

The in-memory distribution processing may include changing the data stored in the external memory to an RDD which is an elastic distributed data set that is in-memory data.

In the generating of the spatial web object, the spatial web RD module includes the spatial information and the word information in the text data, which is the Aldi, generated by the in-memory distribution processing unit, respectively. A spatial web CD generating step of generating a CD; A grid caddy generation step of generating a grid aldi (GriddedRDD) module by dividing the spatial web aldi by a space partitioning algorithm and constructing a partitioned individual space into one partition; And an IndexeddiD process of generating an Indexeddid by which an IndexedDid module constructs an index for searching for a spatial web object included in each partition of the griddiddy.

In addition, the spatial web object querying step, a Boolean Range Query module receives the spatial information and the word information, the range query for searching for documents containing all the words of the spatial web object in the corresponding spatial range fair; And a Boolean Top-K Query module receiving the spatial information and the word information and searching for the K closest spatial web objects containing all of the words at the corresponding spatial location. can do.

As described above, according to the distributed processing-based spatial web object search system and the distributed processing-based spatial web object search method using the same, the distributed processing system is efficiently used by using spatial partitioning, and the partitioned space is searched. There is an effect to increase the efficiency of.

In addition, the present invention has an effect that can show excellent performance in the search of the spatial web object by applying the QP-tree to build the index by using both the spatial information and the word information for the data search.

1 is a block diagram of a distributed processing-based spatial web object search system according to an embodiment of the present invention.
2 is a block diagram of a spatial web object generating unit according to the present invention.
FIG. 3 is a diagram illustrating a GridDID generation algorithm of the GridDRD module according to the present invention.
FIG. 4 is a diagram illustrating an index aldehyde generating algorithm of the IndexedRDD module according to the present invention.
5 is a block diagram of a spatial web object query unit according to the present invention.
6 is a diagram illustrating a query performing algorithm of a Boolean Range Query module according to the present invention.
7 is a block diagram of a distributed processing-based spatial web object search method according to an embodiment of the present invention.
8 is a block diagram of a step of creating a spatial web object according to the present invention.
9 is a block diagram of a spatial web object query step in accordance with the present invention.
10 illustrates a comparison of Boolean Range Query performance.
11 is a diagram illustrating a comparison of query performance by data amount.
12 shows a comparison of Boolean Top-k Query performance.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention; First, it should be noted that the same components or parts in the drawings represent the same reference numerals as much as possible. In describing the present invention, detailed descriptions of related well-known functions or configurations are omitted in order not to obscure the gist of the present invention.

1 is a block diagram of a distributed processing-based spatial web object search system according to an embodiment of the present invention.

As shown in FIG. 1, the distributed processing based spatial web object search system according to the present invention includes an in-memory distributed processing unit 100, a spatial web object generating unit 200, and a spatial web object querying unit 300.

The in-memory distribution processor 100 may change the data stored in the external storage device into in-memory data used on the memory.

The in-memory distributed processing unit 100 may store data stored in an external storage device including a high-availability distributed object-oriented platform (Hadoop) or a local disk, an in-memory-based distributed processing system, that is, an in-memory usable in Spark. You can change the data type to an elastic distributed dataset, or Resilient Distributed Dataset (RDD).

In detail, the in-memory distributed processing unit 100 may change the corresponding data into a raw RDD, and the raw RDD may store one raw data of a spatial web object in one line.

The spatial web object generator 200 may generate spatial web objects by including spatial information and word information in the in-memory data changed by the in-memory distribution processor.

2 is a block diagram of a spatial web object generating unit according to the present invention.

Specifically, as shown in FIG. 2, the spatial web object generating unit 200 includes a spatial web RD module 210, a grid RDD module 220, and an index RDD module. 230 may be included.

The spatial web RDD module 210 may generate spatial web aldi as a set of spatial web objects by including spatial information and word information in text data generated by the in-memory distributed processor, respectively. .

The gridded RDD module 220 may divide the space web Aldi using a space dividing algorithm and generate grid dividing each divided space into one partition.

The GridDRD module 220 may increase the efficiency of searching by dividing a spatial area of the entire data into a global grid and allocating data belonging to one grid to one partition.

Specifically, the GriddedRDD module 220 may perform spatial partitioning of the entire area using the STR algorithm or the BSP algorithm, which is a spatial partitioning algorithm, and then divides data belonging to each partitioned area into <gridid, spatialwebobject. >, You can sort data by gridid, and then reorganize partitions by group.

FIG. 3 is a diagram illustrating a GridDID generation algorithm of the GridDRD module according to the present invention.

The gridded RD module 220 may generate a grid CD using an algorithm as shown in FIG. 3.

Specifically, the GriddedRD module 220 may generate a text file, which is a raw file, as a rawRDD file, through a spark.textFile function.

After that, the generated rawRDD can be transformed into swoRDD, which is an RDD that stores spatial web objects through the map function, which is a function of Spark, and grid division can be performed through the genGrids function. After receiving the type of partition algorithm, the sample rate σ, and the number of partitions β, the input RDD is divided into spaces by applying a partitioning algorithm, and the set of partitioned grids can be returned in the form of an ArrayList.

On the other hand, when grid division is completed, griddedRDD can be created by using mapToPair function in swoRDD. This mapToPair function converts data in RDD into a pair of data. ) Take a single object and judge the inclusion relationship with the grid in girds. When the spatial location of swo is included in the grid area, make the swo a tuple of <gridid, swo> type, and <gridid, swo> type. It can be put in griddedRDD which is the RDD of.

In other words, after the mapToPair function, griddedRDD is composed of rdd, which is a pair of grid IDs per spatial web object, and since the data size of griddedRDD is larger than the raw data, it is necessary to repartition the partition.

Therefore, partitionBy can be executed to create as many partitions as the number of grids. If the number of partitions is changed, groupByKey can be run on griddedRDD to create groups for each grid to create groupRDD. PairRDD, which can store pairs of, may consist of <gridid, set of swo>.

The IndexedRDD module 230 may generate an Indexeddid that constructs an index for searching for a spatial web object included in each partition of the Griddidi.

Specifically, the IndexedRDD module 230 may build one index in one partition, and may also use a QP-tree as an index of the spatial web object. The entire data is formed into a quad-tree using spatial information of the spatial web object, and the end node of the quad-tree may be configured as a Patricia-trie for quick retrieval of word information belonging to the node.

FIG. 4 is a diagram illustrating an index aldehyde generating algorithm of the IndexedRDD module according to the present invention.

The IndexedRDD module 230 may generate an Indexeddid using an algorithm as shown in FIG. 4.

In detail, the mapPartitions function is executed for each partition of the groupRDD generated in the griddedRDD step. When the function is executed, the mapPartitions function may generate the index, that is, the QP-tree, by taking the spatial area information (MBR) of the current partition.

Then, you can build an index by taking the spatial web objects in the current partition and inserting them into index. The tuple._1 () function gets the first value of the current tuple.

After index construction is completed, <gridid, index> data can be put in the result list as the result value.When the mapPartitions function is finished, indexedRDD becomes RDD with data pairs of grid ID and index. You can run the persist function, which is used to store things in memory for faster processing.

The spatial web object query unit 300 searches a spatial web object including the corresponding spatial information and the word information by performing a query on the spatial web object generated by the spatial web object generating unit for specific spatial information and word information. Can be.

5 is a block diagram of a spatial web object query unit according to the present invention.

In detail, as illustrated in FIG. 5, the spatial web object query unit 300 performs a Boolean Range Query module 310 and a Boolean Top-k Query module 320. It may include.

The Boolean Range Query module 310 may receive spatial information and word information and search for a document including all of the corresponding words among the spatial web objects within the corresponding spatial range.

Specifically, when the user inputs spatial information (location information) and word information, the Boolean Range Query module 310 has an index (QP-tree) that each RDD has using the input spatial information. ) Can be used to query the spatial information.

After that, when the query is executed, it is possible to check whether the RDD corresponds to the query by using the MBR of the root. Therefore, if the RDD does not correspond to the query, the next RDD can be fetched without executing a further search and the RDD corresponding to the query. In this case, the search may be performed in the QP-tree using spatial information of the query.

If you search the terminal node by QP-tree, you can search for the object containing the word entered by the user in Patricia-trie included in the terminal node and return it as a result.

6 is a diagram illustrating a query performing algorithm of a Boolean Range Query module according to the present invention.

That is, as shown in FIG. 6, the Boolean Range Query module 310 first receives an indexedRDD configured as an index, and uses information (position, range, and words) input by the user to the indexedRDD. A query can be performed on the indexes in each partition, and the result of the query execution is stored in the result list, which is the final result, and returned as a result. The results performed in each partition can be aggregated in master.

The Boolean Top-k Query module 320 may receive spatial information and word information, and search for the nearest spatial web object K including all of the words at the corresponding spatial location.

Specifically, when the user inputs spatial information (location information) and word information, the Boolean Top-k Query module 320 performs a BkQ query for each RDD using the input information. The Boolean Top-k Query module 320 retrieves k objects using a priority queue from each index in the RDD, and the searched results can be passed to the master as a result. From the number of temporary results obtained from k * partitions, the priority queue can be used again to find the K search results with the highest priority.

Hereinafter, a distributed processing based spatial web object searching method according to the present invention will be described.

7 is a block diagram of a distributed processing-based spatial web object search method according to an embodiment of the present invention.

As illustrated in FIG. 7, the distributed processing-based spatial web object search method includes an in-memory distributed processing step S10, a spatial web object generation step S20, and a spatial web object query step S30. .

The in-memory distribution processing step S10 is a step in which the in-memory distribution processing unit 100 changes the data stored in the external storage device into in-memory data used on the memory.

In detail, in the in-memory distributed processing step S10, the in-memory distributed processing unit 100 may change data stored in an external storage device into an RDD, which is an elastic distributed data set that is in-memory data.

The spatial web object generation step (S20) is a step in which the spatial web object generation unit 200 includes spatial information and word information in the in-memory data changed by the in-memory distribution processing unit to generate the spatial web object, respectively.

8 is a block diagram of a step of creating a spatial web object according to the present invention.

Specifically, as shown in FIG. 8, the spatial web object generating step (S20) may include a spatial web CD generating process (S21), a grid CD generating process (S22), and an index CD generating process (S23). Can be.

The spatial web CD generating step (S21) is a spatial web RD module 210 is a set of spatial web objects by including the spatial information and word information, respectively, in the text data generated by the in-memory distributed processing unit Aldi The process of creating a space web Aldi.

Grid grid generation step (S22) is a grid grid (RDID) module (220) grid partition the space Web Aldi using a space partitioning algorithm and each partitioned space to build a grid It is a process to produce an Aldi.

Specifically, in the grid CD generation process (S22), the spatial web CD may be spatially partitioned using the STR algorithm or the BSP algorithm, which is a spatial partitioning algorithm.

In the index eddie generating step (S23), the indexed eddy module 230 generates an indexed eddy that constructs an index for searching for a spatial web object included in each partition of the grid aldi. .

In detail, in the step S23 of generating an index aldehyde, a QP-tree may be used as an index of the spatial web object.

The spatial web object querying step (S30), the spatial web object query unit 300 queries the spatial web object generated by the spatial web object generating unit for specific spatial information and word information so that the corresponding spatial information and word information are stored. Searching for the included spatial web object.

9 is a block diagram of a spatial web object query step in accordance with the present invention.

In detail, as illustrated in FIG. 9, the spatial web object querying step S30 may include a range querying process S31 and a pyramid kay querying process S32.

In the Boolean range query step S31, a Boolean Range Query module 310 receives a spatial information and word information and searches for a document including all of the words among the spatial web objects within the corresponding spatial range. to be.

In step S32, the Boolean Top-K Query module 320 receives the spatial information and the word information, and includes the nearest spatial web object K including all the words at the corresponding spatial location. The process of searching for dogs.

Hereinafter, an example of the performance experiment of the spatial web object search method using the distributed processing-based spatial web object search system according to the present invention will be described.

In this experiment, the actual Twitter data was used, and the amount of data was 30 million, which required about 5GB of storage space, and the device used for the experiment consisted of one master and six workers.

The Master has 12GB of RAM in the Intel Xeon e5540 (4 cores), and the Walker has 16GB of RAM in each Intel i5 3570 (4 cores). In fact, Spark used a system with 12GB of RAM per worker, with a total of 24 cores and 72GB of RAM.

Experimental data were used in the experiment by making 5 million, 10 million, 15 million, and 25 million data, respectively. Also, in this experiment, a method using R-tree, which is an index using only conventional spatial information, for comparison of search performance ( Spatial technique) and QP-tree, which is an index that uses both spatial and word information of the present invention, (spatial + word technique).

<BRQ performance comparison>

In this experiment, we compared the performance of BRQ query, one of spatial web object queries. The point information, which is a parameter used in the experiment, was randomly extracted from the entire data, and the word information was scanned using the whole word, the word with the highest frequency, the word with the medium frequency, and the word with the lowest frequency, respectively. The query was repeated 10 times and the average of query execution time was calculated.

The data used in the experiment was 25 million data, and the search range was increased by 10 times to 0.001%, 0.01%, 0.1%, 1%, and 10% of the entire range to measure the search performance.

10 illustrates a comparison of Boolean Range Query performance.

As a result of the experiment, as shown in Figure 10, it can be seen that the method using the space + words as shown in the present invention shows a much better performance than the method using the conventional space only.

This is because the spatial technique uses space to search for data and then search for additional words, so it takes a long time to perform the secondary search using the results of the primary search result (spatial search). Since the word technique can search for spaces and words in the index in indexedRDD, it can show almost constant search performance regardless of space size.

<Query Performance Comparison by Data Volume>

In this experiment, we confirmed the query execution time by changing the amount of data in the BRQ query. The data was changed to 10 million, 15 million, 20 million and 25 million, and the search range used in the experiment used 0.1% of the total data.

11 is a diagram illustrating query performance comparison by data amount.

As a result of the experiment, as shown in FIG. 11, the spatial + word scheme according to the present invention has almost no change in search time according to the size of the data, whereas the conventional spatial technique takes a longer search time as the size of the data increases. .

This is because the spatial technique increases the size of the data, increases the amount of data in the same range, and increases the amount of objects to be searched, which increases the data for the words to be searched.

On the other hand, in the space + word technique, even if the amount of data increases, since the index for the word information in the same node is constructed, the search time is hardly changed.

<BkQ Performance Comparison>

In this experiment, we compared the performance of BkQ queries, one of spatial web object queries. The data used in the experiment were 25 million Twitter data, and the location of the query was randomly extracted from all the data. Also, the K value was increased by 10, 20, 30, 40, and 50, and the performance of the query was tested.

12 shows a comparison of Boolean Top-k Query performance.

As a result of the experiment, as shown in FIG. 11, the conventional spatial technique showed similar performance in the vicinity of 350 ms, and the spatial + word technique according to the present invention showed similar performance in 100 ms.

This means that the spatial technique takes time to retrieve the data from the R-tree belonging to one partition, and then adds a word search from the retrieved data, while the spatial + word technique uses the index from one partition (QP-tree). When you do a search, you can search for information that includes words, which can take less time.

As described above, according to the present invention, it is possible to efficiently use a distributed processing system by using spatial partitioning, to improve the efficiency of searching for partitioned space, and to use spatial information and word information together to search data. By applying the QP-tree to build indexes, we can show excellent performance in searching spatial web objects.

As described above with reference to the drawings illustrating a distributed processing-based spatial web object search system and a distributed processing-based spatial web object search method according to the present invention, the present invention by the embodiments and drawings disclosed herein Of course, various modifications may be made by those skilled in the art within the scope of the technical idea of the present invention.

100: in-memory distribution processing unit
200: spatial web object generation unit
210: SpatialWebRDD module
220: GriddedRDD module
230: IndexedRDD module
300: spatial web object query unit
310: Boolean Range Query Module
320: Boolean Top-k Query Module
S10: In-Memory Distributed Processing Steps
S20: Creating a Spatial Web Object
S21: Spatial Web CD Creation Process
S22: Grid Aldi production process
S23: Index Aldi generation process
S30: Spatial Web Object Query Step
S31: Poor quality process
S32: Firetop Cayman's Process

Claims (10)

An in-memory distribution processor for changing data stored in the external storage device into in-memory data used on the memory;
A spatial web object generating unit generating spatial web objects by including spatial information and word information in the in-memory data changed by the in-memory distribution processor; And
And a spatial web object query unit which searches a spatial web object including corresponding spatial information and word information by querying the spatial web object generated by the spatial web object generator for specific spatial information and word information.
The spatial web object generated by the spatial web object generator is a document including information for confirming information about a region.
The spatial web object generation unit,
A Spatial WebRDD module for generating a spatial Web RDI that is a set of spatial web objects by including spatial information and word information in the Aldyin text data generated by the in-memory distributed processor;
A Gridded RD module for generating a grid RD by dividing the spatial web Aldi using a space dividing algorithm and constructing each divided space into one partition; And
And an IndexedRDD module for generating an indexed Aldi for building an index for searching for a spatial web object included in each partition of the grid Aldi.
The IndexedRDD module uses a QP-tree as an index of the spatial web object.
The QP-tree is a quad-tree using the spatial information of the spatial web object, and the end node of the quad-tree consists of a Patricia-trie for quick retrieval of word information belonging to the node. Distributed web based object retrieval system based on distributed processing.
The method of claim 1,
The in memory distribution processing unit,
The distributed processing-based spatial web object retrieval system according to claim 1, wherein the data stored in the external storage device is changed into a resilient distributed dataset (RDD), which is an in-memory data.
delete The method of claim 1,
The Gridded RD module,
Distributed spatial processing based spatial web object retrieval system, characterized in that for performing the spatial partitioning using the STR algorithm or BSP algorithm, which is a spatial partitioning algorithm.
delete The method of claim 1,
The spatial web object query unit,
A Boolean Range Query module for receiving the spatial information and the word information and searching for a document including all of the corresponding words among the spatial web objects within the corresponding spatial range; And
A Boolean Top-k Query module for receiving the spatial information and the word information and searching the K nearest spatial web objects containing all of the words in the corresponding spatial location. Based spatial web object retrieval system.
A distributed processing based spatial web object searching method using a distributed processing based spatial web object searching system according to any one of claims 1, 2, 4 and 6,
An in memory distribution processing step of changing, by the in memory distribution processing unit, data stored in the external storage device into in memory data used on the memory;
A spatial web object generation step of generating a spatial web object by including spatial information and word information in the in-memory data changed by the in-memory distributed processing unit; And
A spatial web object query step of searching for a spatial web object including the spatial information and the word information by performing a query on the spatial web object generated by the spatial web object generating unit by the spatial web object generating unit for specific spatial information and word information; Distributed processing-based spatial web object search method comprising a.
The method of claim 7, wherein
The in memory distribution processing step,
The distributed memory-based spatial web object retrieval method according to claim 1, wherein the in-memory distributed processing unit changes data stored in an external storage device into an RDD which is an elastic distributed data set that is in-memory data.
The method of claim 8,
The spatial web object creation step,
A spatial web eddy generating step of generating a spatial web aldi as a set of spatial web objects by including spatial information and word information in the text data generated by the in-memory distributed processing unit;
A grid caddy generation step of generating a grid aldi (GriddedRDD) module by dividing the spatial web aldi by a space partitioning algorithm and constructing a partitioned individual space into one partition; And
An indexeddidi generating step of generating an indexeddid by which an indexeddid module constructs an index for searching a spatial web object included in each partition of the griddidi; How to retrieve based spatial web objects.
The method of claim 9,
The spatial web object query step,
A Boolean range query step of receiving, by a Boolean Range Query module, spatial information and word information and searching for a document including all of the corresponding words among spatial web objects within the spatial range; And
The Boolean Top-K Query module receives the spatial information and the word information, and searches the K- Nearest Spatial Web object containing all the words at the spatial location. Distributed processing based spatial web object search method, characterized in that.

Images