KR20140076010A - A system for simultaneous and parallel processing of many twig pattern queries for massive XML data and method thereof - Google Patents

Abstract

The present invention relates to a system for processing multiple twig pattern queries on massive XML data simultaneously and parallelly and a method thereof. The system includes: an input unit which receives, from users, an input of twig pattern queries and a massive XML file; a pre-processing and data loading unit which divides the twig pattern queries inputted through the input unit into linear path patterns, creates a query index using the linear path patterns, creates multiple XML data blocks from the massive XML file, and loads the created XML data blocks to a distributed file system; a first map-reduce work unit which receives the XML data blocks to obtain answers to the linear path patterns, and calculates the size of the answers to the linear path patterns; and a second map-reduce work unit which performs twig pattern join operations using the answers to the linear path patterns to output the final results.

Description

Technical Field [0001] The present invention relates to a system and method for simultaneous parallel processing of multiple pattern queries on large-capacity XML data,

The present invention relates to a query processing system for XML data and a method thereof. And more particularly, to a method for simultaneously distributing / paralleling multiple patterns of pattern queries on very large capacity XML data that is difficult to process in a reasonable time on a single node. The present invention also relates to a multi-query optimization method and an XML document division processing system and method thereof for efficiently distributing / parallel processing a plurality of branch pattern queries.

XML (eXtensible Markup Language) has a semi-structured data model as a data storage format that provides not only content information but also structure information of data using tags. The data described in XML represents the relationship between the tags according to their inclusion relation. For example, when using XML to represent book information, there is an element named "book" in XML data, and the child elements of the element are "title", "author" and "publisher" , And elements that are required for book information presentation can be structured in such a way that the child elements of "author" include elements named "first name" and "first name". In order to process the query of XML data including the relation of such information, the query information for XML may include structure information. For example, in the previous example, the relationship of each element can be expressed in path patterns such as "/ book / author" or "/ book // publisher". The extraction of information with a desired relationship using this path expression is called a linear path query. In addition, you can query these more complex patterns by grouping these various linear path queries. For example, a query to list the titles of a book published by Minumsa, "/ book [./ publisher =" minsumsei "], combines the conditional path expression" / book / title " Publisher = "Minminsa"] / title ". This type of query is called a twig pattern query because it has a kind of relationship like a twig. This branch pattern is the most frequent form of query pattern for XML data. The query processing of such a branch pattern is generally performed by 1) decomposing branch patterns into linear path patterns, 2) processing each linear path pattern query to obtain the result, and 3) joining the results of these linear path pattern queries join).

The holistic twig pattern join method is a join method designed to perform the query processing of the branch patterns optimally in terms of input and output. This method is a method of excluding the result of the query of the path pattern that can be actually joined and obtained as a final result when the result of the linear path pattern query is obtained. This method is also optimized for I / O because it can read the lists of each element sequentially one time only and get the final result. In the present invention, this holistic branch pattern joining technique is used to process each branch pattern query.

XML stream processing is a series of processing of a series of small XML documents (several to several tens of KB) in real time. Stream processing is not limited by the number of data as the data is continuously generated from various data sources, Can be prepared in advance, and is a task having a characteristic that a query process for newly generated stream data must be performed in real time.

YFilter processes XML stream data with a technique disclosed in a paper of "Path sharing and predicate evaluation for high performance xml filtering, Y. DIAO et al., ACM Translations on Database System, 28 (4): 467-516, 2003." (NFA: Nondeterministic Finite Automata) to share partial paths that are common to several linear path patterns. This technique is based on a stream environment in which small size XML data is input as continuous input, and it guarantees fast filtering speed based on memory. However, in the present invention, a technique for processing a plurality of XML path query processing for stream data in which XML documents of a small size of several tens to several hundreds KB are successively received in the memory of a single node, There is a problem in that it is not possible to process a large-capacity XML document which is more than a predetermined size.

Accordingly, a method of dividing a large-capacity XML document into blocks of the same size without loss of structure information of a document, a function of performing a plurality of path pattern queries on the block, and a function of querying a plurality of branch pattern queries There is a need for an invention on an efficient distributed processing method of a processing operation.

US Patent No. 7,756, 919 (filed on June 16, 2004) entitled " Large-scale data processing in a distributed and parallel processing environment " is an invention for parallel processing of data records named MapReduce. According to the present invention, a programming model for dividing an input file into a plurality of blocks, reading data in each block as a key / value pair and processing them with Map and Reduce functions, and a distributed / To a parallel processing method.

MapReduce is a S / W framework designed to improve the performance of web search system by efficiently analyzing web page information of early large capacity and enables large amount of data stored in distributed file system to be processed in parallel in distributed computing environment . The MapReduce is divided into fixed size, each piece of distributed stored files is read by each computing node into a key / value pair, and the Map function is executed to generate intermediate results in the form of another key / value pair. The intermediate results are then grouped based on the key values and then aggregated through the Reduce function to produce the final result. Hadoop is open source software written in accordance with this paradigm, and the overall system configuration is the same as the original MapReduce.

However, the patented invention divides the data into key and value pairs, divides the file into fixed sizes without considering the structure information of the document, and devises only a single query processing. There is a limit to parallel processing multiple branch pattern queries simultaneously.

Therefore, it is possible to greatly reduce the input / output cost by applying a technique of dividing and parallelizing large-capacity XML files without loss of structure information and sharing common single-path solutions among the branch patterns, There is a need for an invention capable of increasing the efficiency of data analysis by performing branch pattern queries in parallel in a distributed environment with a two-step maple deuce operation.

The present invention provides a distributed / parallel processing technique based on MapReduce for processing a plurality of branch pattern queries. However, there are the following problems when processing the branch pattern query for XML with MapReduce at this time.

1) In MapReduce, a file is divided and stored into blocks of fixed size, and data in the block is read and processed in the form of key / value pairs. However, when an XML file is simply divided into a plurality of file fragments at a fixed size, it is impossible to process the query as the relationship information between the elements, which is XML-specific structure information, is lost.

2) When a single branch pattern query is implemented as one mapping process, n map re-deuce processes are performed to perform n query processes, which perform similar operations repeatedly and repeat the same data Is inefficient. Therefore, there is a need for a method that can process many pattern queries at a single time with as few mapping processes as possible.

3) Many branch pattern queries have many parts that are shared with each other. Therefore, there is a need for a method for eliminating the repetitive processing of common patterns by sharing the results of the intermediate results among the queries for fast processing.

4) By processing multiple branch pattern queries, multiple branch pattern join operations must be executed for multiple nodes. At this time, it is possible to obtain the fastest query processing time by distributing the join operations to a smaller number of nodes in a balanced manner and ending each node at the same time with the same amount of work. At this time, there is a need for a solution to how to distribute branch pattern queries equally to each node.

U.S. Patent No. 7,756,919 (filed on June 18, 2004)

Path sharing and predicate evaluation for high performance xml filtering, Y. DIAO et al., ACM Translations on Database System, 28 (4): 467-516, 2003.

It is an object of the present invention to provide a method and apparatus for efficiently processing a large number of pattern queries on large capacity XML data, ) Is continuously generated and periodically updated. Therefore, it is required to perform fast query processing. In order to deal with such large amount of XML data, many pattern queries are simultaneously distributed / parallelized System and method therefor.

According to a first aspect of the present invention, there is provided an information processing apparatus comprising: an input unit for receiving branch pattern queries and a large-capacity XML file from users; A branching unit for branching the branch pattern queries input into the input unit into linear path patterns, generating query indexes using the branch pattern queries, and dividing the large capacity XML file into fixed size block units for parallel processing in a plurality of computers, A preprocessing and data loading unit for creating XML data blocks of the XML file and loading the XML data blocks into a distributed file system; A first mapping task unit for receiving the divided XML data blocks and obtaining a solution to the linear path patterns at a map step using a query index and calculating a size of a solution for each linear path patterns at a reducing step, Wow; In this paper, we propose a new algorithm that can distribute the branch pattern join operations to multiple computing nodes in real time with equal workload, and solve the linear path solutions that they will take as input for the branch pattern join operations assigned to the various nodes, And a second mapping task unit for performing branch pattern join operations allocated to each reducer in the redist step with the solutions of the linear path patterns received in the map step and outputting the final result 104. [ A concurrent parallel processing system for multiple branch pattern queries is presented.

According to a second aspect of the present invention, there is also provided a method of searching for a document, the method comprising: receiving branch pattern queries and a large-capacity XML file from users; Decomposing the branch pattern queries inputted into linear path patterns; Generating a query index using the branch pattern queries decomposed into the linear path patterns; Generating a plurality of XML data blocks by dividing the large capacity XML file into blocks of a fixed size so as to enable parallel processing in a plurality of computers; Loading the plurality of XML data blocks into a distributed file system; Obtaining a solution to the linear path patterns in the mapping process of the first mapping task unit using the query index by receiving the divided XML data blocks; Calculating a size of a solution for each linear path pattern in a process of reducing the size of the first mapping task; Distributing the branch pattern joining operations to the plurality of computing nodes in real time evenly with respect to each other; Transmitting the solutions of the linear paths to be taken as input to the branch pattern join operations assigned to the plurality of nodes, at the execution time in the mapping process of the second mapping task unit; Performing branch pattern join operations allocated to each reducer in the redistribution process of the second mapping task unit with the solutions of the linear path patterns received in the mapping process and outputting the final result; A parallel processing method of branch pattern queries is presented.

The present invention relates to a method for efficiently processing a large number of query terms in a distributed environment for large-capacity XML data, in particular, a large-capacity XML file, and a plurality of different pattern queries are simultaneously distributed / There is an effect that can be done. In addition, it is possible to share inputs common to multiple branch pattern queries and their results at the time of query processing, and to eliminate redundant processing by replacing path expressions containing double slashes and wildcards with unique path patterns in the document, The input / output cost is reduced, and the efficient query processing can be performed. Also, by applying a technique that balances the workload of the join operation for processing the branch pattern between distributed nodes, it is possible to prevent performance degradation of the whole system due to a bottleneck that the workload is biased toward one node, There is an effect that can be performed. Particularly, the task allocation technique of the present invention is not performed statically before the query processing but measures the size of the intermediate result of the query processing, that is, the solution for each path pattern after the XML data is inputted, .

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic block diagram of an embodiment of a simultaneous parallel processing system for a plurality of different pattern queries for large capacity XML data of the present invention.
FIG. 2 is a schematic block diagram of an embodiment of a preprocessing and data loading unit which is a main part of the embodiment of the present invention.
3 is a schematic diagram of an embodiment of a function of converting a large capacity XML file as an input into a fixed size block unit without loss of structure information between XML elements and locating the corresponding label file in a distributed file system according to the present invention FIG.
4 is a schematic explanatory diagram of an embodiment relating to the operation of the first mapping unit of the embodiment of the present invention.
5 is a schematic explanatory diagram of an embodiment of the operation of the second mapping task unit of the embodiment of the present invention.
6 is a flowchart for explaining an embodiment of a detailed operation process of the multiple query optimization module in the embodiment of the present invention.
FIG. 7 is an explanatory diagram of a process of converting a linear path pattern in which a double slash and a wildcard exist, into unique path patterns existing in an input XML document according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic block diagram of an embodiment of a simultaneous parallel processing system for a plurality of different pattern queries for large capacity XML data of the present invention. As shown in FIG. 1, the simultaneous parallel processing system for a large number of pattern queries of large capacity XML data according to the present invention includes an input unit 100 for receiving branch pattern queries and a large capacity XML file from users; A query index 101 is generated by decomposing the branch pattern queries input to the input unit 100 into linear path patterns and using them to parallelize the large XML file in a plurality of computers, A preprocessing and data loading unit (110) for creating a plurality of XML data blocks (102) by dividing into blocks and loading them into a distributed file system; A solution 103 for the linear path patterns is obtained in the map step using the query index 101 by receiving the divided XML data blocks 102 and the size of the solution for each linear path pattern is determined in the reducing step A first mapping task unit 120 for calculating the first mapping task; In this paper, we propose a method to solve the linear pattern path problem by applying the linear pattern path join operations to the multiple computing nodes in real time. In the redist step, branch pattern join operations assigned to the respective reducers are performed with the solutions 103 of the linear path patterns received in the map step and the final result 104 is outputted (130).

2 is a schematic block diagram of an embodiment of the preprocessing and data loading unit of the embodiment of the present invention. A query division module 200 for dividing each branch pattern query into a plurality of linear path patterns constituting the branch pattern query and generating correspondence information between the linear path patterns and the original branch pattern query, ; A query index creation module 202 for generating a query index composed of nondeterministic finite automata with the linear path patterns; The XML file is input as a large capacity XML file and the XML files are divided into fixed size XML data blocks without loss of structural information and the label values assigned according to the area based numbering technique are listed for the XML elements and attributes belonging to each data block And an XML data segmentation and labeling module 201 for collecting and loading the label blocks into a distributed file system.

The preprocessing process includes preprocessing by the query division module 200 and the query index creation module 202 for the branch pattern query previously input from the plurality of users before the query processing and the XML data segmentation and labeling module for the large- (2).

The query partitioning module 200 may, for example, provide T1: / A / B [C / D] / E / F [G] / H, (P1, P2, P3) with three linear path patterns: P / B / E / F / H, P2: / A / B / C / D and P3: / A / B / Then, T1: == {P1, P2, P3}, which is correspondence information between them, is also extracted and stored separately. In this way, all branch pattern queries are divided into linear path patterns.

The query index generated by the query index creation module 202 is used to perform filtering on the linear path patterns with respect to the XML data blocks in the first mapping task unit 120. In the present invention, an existing technique called YFilter is used for generating the query index, but the present invention is not limited thereto. For example, other XML stream processing techniques that perform XML filtering can be used. For query processing, we first perform filtering with linear path patterns, and then pattern join the results of these linear path patterns. The reason why the branch pattern queries are divided into the linear path patterns and the query processing is performed first is that there are linear path patterns common to the various pattern queries. In case of separately processing them, This is because the repetitive query processing on the path patterns increases the amount of input and output and the processing time. Processing only the linear path patterns for all the branch pattern queries during the query process reduces the size of the intermediate result between the first and second mapping tasks 120 and 130, This is to save network and disk I /

In the embodiment of the XML data segmentation and labeling module 201, the size of the XML data block size is set to 64 MB, but this size can be arbitrarily changed by the user for performance optimization. In addition, the XML data segmentation and labeling module 201 co-locates two blocks corresponding to each other in an XML block and a label block in one distribution node. The reason for doing this is to secure the spatial locality because the first mapping work unit 120 must read the same XML block and label block at the same time. It can be expected that the network input / output cost is reduced due to the secured spatial locality.

3 is a schematic diagram of an embodiment of a function of converting a large capacity XML file as an input into a fixed size block unit without loss of structure information between XML elements and locating the corresponding label file in a distributed file system according to the present invention FIG.

First, when the original XML file 300 is divided into the XML data blocks 301, basically, it is divided into a given fixed size reference. However, if the element itself inside the XML file is adjacent to the boundary between two blocks and the element itself is cut off in the middle, the element can not be distinguished by the tag name. Therefore, in the XML data segmentation of the present invention, as long as the element is not truncated, it is divided into several XML data blocks based on a given block size. In the case of the XML data block divided according to the block division, the path information of the preceding blocks is stored in the label block 302 corresponding to the XML data block as the structure information of the preceding blocks is lost. Label block 1 (302) is the first block and thus indicates / indicating the root. Label block 2 is a label block corresponding to the second XML data block, and the path information of the first element of the second XML block </ D> In / A / B / C / D. When querying the XML data block, the structure information of the first element of the XML data block is restored by using the path information at the beginning of the corresponding label block. The three numbers that are displayed later in the label block 302 are the numbers assigned to each element and the attribute according to the area-based numbering scheme, and the start and end of the area represented by the element, respectively, Quot; means a level value to be positioned. In the area-based numbering technique, if the area start value of the XML element is smaller than the area start value of B and the area end value of A is larger than the area end value of B, A means that the parent has an ancestor relation of B, A value of 1 means parent-child relationship, and 2 or more means ancestor-descendant relationship.

FIG. 4 is a schematic configuration diagram of an embodiment of a first mapping task unit 120 according to an embodiment of the present invention. 4, the first mapping task unit 120 receives the XML data blocks and the label blocks divided and stored in the preprocessing and data loading unit 110 and inputs a query index 204 To process linear path patterns. The map execution nodes 400 of the first mapping task 120 load the query index 402 first and read them from the distributed file system with the XML block and the corresponding label block as inputs. In this case, the query indexes used in each map step have the same structure. In each map step, each XML block is filtered using this query index, and the label value corresponding to the solution of the linear path patterns included in the query index is output to the output. At this time, the format of the output is conceptually (the ID of the linear path pattern, the label value). At the node 401, the size information 403 of the linear path solutions is separately recorded by grouping the linear path solution labels according to the IDs of the linear path patterns.

5 is a schematic configuration diagram of an embodiment of the second mapping task 130 of the embodiment of the present invention. As shown in FIG. 5, the second mapping task unit 130 performs pattern join operations by inputting the linear path solutions 404 obtained from the first mapping task unit 120, and outputs the final result Calculation, and output. However, before the first mapping task is completed and before the second mapping task is started, a multiple query optimization module 502 is performed to previously distribute the branch pattern joins to the nodes evenly with respect to each other. The concrete operation contents of the multi-query optimization module 502 will be described later in detail with reference to FIG. The multi - query optimization module distributes the pattern - join operations to each redistribution node so that the workloads are equal to each other. The map execution node 500 performs a redescription operation in which a branch pattern operation for inputting a solution of linear path patterns as input is performed based on information about which branch pattern operation has been distributed to which node To the node 501. This process is performed by replacing the key value of the map execution result with the ID of the redissued node to be transmitted.

In this case, when a solution of a linear path pattern is transferred to a plurality of redundancy execution nodes, replication of the solution of the linear path pattern is performed, thereby increasing network input / output cost. . The node performing the redundancy pattern receives the solution of the linear path patterns received and performs branch pattern join operations assigned to the node, thereby calculating and outputting the final result.

6 is a flowchart for explaining an embodiment of a multiple query optimization module according to an embodiment of the present invention. As shown in FIG. 6, step (S100) of reading correspondence relationship information between the branch pattern queries and the linear path patterns; Reading the size information of the solutions of the linear path patterns read (S101); Calculating the cost of each branch pattern join operation based on the size of the linear path pattern solution (S102); (S103) searching for a branch pattern query having the largest cost as a result of the calculation, and if there is a linear path common to the already allocated branch pattern, eliminating it from the cost by the size of the linear path; A step (S104) of allocating the node to a redundancy performing node having no assigned branch pattern query; A step (S105) of judging whether there is a redundancy-executing node to which a branch pattern query is not allocated; If there is a redundancy node to which the branch pattern query is not allocated, the process returns to step S103. If there is no redundancy node to which the branch pattern query is not allocated, the cost of the allocated branch pattern query is reduced Searching for a node (S106); Finding a branch pattern query having the largest solution size of the linear path pattern common to the allocated branch pattern queries of the redistributed node having the lowest cost of the found branch pattern query, and allocating the same to the corresponding redundancy performing node S107); Determining whether there is an unassigned branch pattern query, and returning to step S107 if there is an unassigned branch pattern query, and terminating (S108) when there is no unassisted branch pattern query.

In the embodiment of the present invention, in the multiple query optimizing module, the redistribution nodes are allocated to branch pattern-matching operations evenly, and a solution of a linear path pattern is assigned to several redistribution nodes, Use a greedy approach to minimize the growing situation.

The multi-query optimization module calculates the expected execution cost of the branch pattern queries using the size information of the solution of each linear path pattern collected and the correspondence information between the linear path patterns and the branch pattern queries. In the embodiment of the present invention, a holistic twig pattern join (HTJ) technique is used as a method for processing a branch pattern query. The cost of a join operation in the HTJ technique is the sum of the sizes of the lists of input elements. Therefore, in the embodiment of the present invention, the sum of the sizes of the solutions of the linear path patterns included in the branch pattern query is the cost of the join operation. However, other techniques for performing branch pattern joining may also be used, depending on the HTJ technique itself for query processing, or not limited thereto. When the cost of each branch pattern join operation is calculated, the pattern query is assigned to each of the redistribution nodes in order of the cost of the join operation. In this case, if a linear path pattern common to the linear path patterns input to the redundant node is input to the redundant node, the amount is subtracted from the linear path solution.

After the pattern join operations are assigned to each of the redistribution nodes, the redistribution node having the least cost of branch pattern join operation is selected. Finds the branch pattern query having the largest size of the linear path solution common to the branch pattern queries assigned to the redistributed nodes and assigns them to the corresponding redistribution nodes. At this time, the branch pattern query to be allocated is limited to the branch pattern query which has not been previously allocated. If an unassigned branch pattern query remains, the above operation is repeated.

7 shows an example of an operation of converting linear path patterns described by XPath including a double slash (//) and a wildcard (*) into a double slash and a unique linear path pattern existing in a wildcardless actual document Fig. The reason for doing this is that if there are double slashes or wildcards in the linear path pattern, the elements in the XML document may appear repeatedly as a solution of multiple linear path patterns repeatedly, resulting in an intermediate result This is because the network and disk I / O costs increase significantly as the size increases and, consequently, iterative processing occurs. In the present invention, in order to solve this problem, the query division module 200 divides the branch pattern queries into linear path patterns, and converts the linear path patterns including double slashes and wild cards into unique path patterns existing in the document, So that all the elements of the XML document are processed by the first mapping task unit 120 at most to the processing result of the linear path pattern.

The conversion process of the path pattern is as follows. The path extracting unit 704 extracts unique path patterns 701 existing in the original XML file during the preprocessing. The extracted unique path patterns are stored in a hash table in reverse order for convenience of search. Then, in the unique path conversion unit 705, the query division module 200 compares the linear path patterns 702 including double slashes and wild cards divided from the branch pattern queries with this hash table to convert them into unique path patterns (703), and sets the correspondence between the original linear path pattern and the linear path pattern unique to the document. This correspondence relationship is used to determine the solution of the linear oblique pattern to perform the branch pattern operation at the map execution 500 of the second mapping job 130. [

In addition, there are double slashes (//) and wildcards (*) in the syntax of XPath which is an XML query language. When these symbols are used in query terms, if one linear path pattern is duplicated in multiple path patterns created by XPath Lt; / RTI &gt; For example, the last element on the linear path A / B / C, C, is a path expression written in XPath // C, // B / C, A / B / C, A // C, A / / C, A / B / *. At this time, if each path expression written in XPath is recognized as separate query patterns, and the query results are handled separately for each path expression, eventually the same C elements are repeatedly extracted, It causes waste.

In order to solve this problem, in the present invention, by replacing a linear path pattern in which a double slash or a wildcard appears with a unique path pattern actually present in an XML document, the elements of the XML data are duplicated as a solution of various linear path patterns, Thereby greatly reducing the input / output costs incurred between each MapReduce operation.

The above-described embodiments of the present invention are only a few of various embodiments of the present invention. It is to be understood that various embodiments included in the technical scope of the present invention fall within the scope of protection of the present invention.

100: Input unit
110: preprocessing and data loading unit
120: First MapReduce Operation Section
130: Second MapReduce work unit
200: query division module
201: XML Data Segmentation and Labeling Module
202: Query Index Creation Module

Claims (11)

An input unit for receiving branch pattern queries and large XML files from users; A preprocessing and data loading unit for generating a query index by using the linear path patterns in the branch pattern queries input to the input unit, generating a plurality of XML data blocks from the large capacity XML file, and loading the XML data blocks into the distributed file system; A first mapping task unit for receiving the XML data blocks and obtaining a solution to the linear path patterns and calculating a solution size for the linear path patterns; And a second mapping task for performing a branch pattern operation on the solutions of the linear path patterns and outputting a final result. The method according to claim 1,
The preprocessing and data loading unit divides the large capacity XML file into blocks of a fixed size and stores the blocks in a distributed file system. The preprocessing and data loading unit divides the divided XML data blocks into a plurality of blocks based on an interval-based numbering scheme Label numbers are assigned to elements and attributes. These label numbers are created in separate blocks and stored in the same computing node as the XML data block. When the large XML file is divided into blocks and stored, the structural information of the XML is not lost The block is divided on the basis of the text value located last in the designated block size, and the linear path pattern from the root of the element appearing first to the next block to be newly divided is converted to a string value as a label number A large amount of XML A plurality of patterns simultaneously parallel processing system inquiry for data. The method according to claim 1,
The first mapping task unit receives the XML data blocks, finds a solution of linear path patterns from each XML data block using the query index in a map step, and finds a solution of each linear path pattern in a reduction step. And the size of the large XML data is calculated. The method of claim 3,
The second map reduction operation unit receives the solutions of the linear path patterns and confirms the branch pattern queries placed at the respective distributed nodes in the map step and outputs the linear path patterns necessary for processing the branch pattern queries to the corresponding node And the final result is stored in the distributed file system after performing the holistic branch pattern matching using the correspondence information between the linear path pattern actually received in the redox step and the linear path pattern-branch pattern query. Concurrent Parallel Processing System for Multiple Patterns Query on Large Capacity XML Data. The method of claim 7,
The second map reduction operation unit receives the solutions of the linear path patterns and confirms the branch pattern queries placed at the respective distributed nodes in the map step and outputs the linear path patterns necessary for processing the branch pattern queries to the corresponding node And the final result is stored in the distributed file system after performing the holistic branch pattern matching using the correspondence information between the linear path pattern actually received in the redox step and the linear path pattern-branch pattern query. Concurrent Parallel Processing System for Multiple Patterns Query on Large Capacity XML Data. The method according to claim 1,
And a multi-query optimization module for performing an operation for distributing work pattern amounts to the nodes evenly after the operation by the first mapping task unit is finished and before the operation of the second mapping task unit is started Wherein the plurality of patterns of the plurality of patterns are arranged in parallel.         Receiving pattern pattern queries and a large capacity XML file from users at an input unit; Decomposing the branch pattern queries input from the pre-processing and data loading unit into linear path patterns; Generating a query index using the branch pattern queries decomposed into the linear path patterns in the pre-processing and data loading unit; The pre-processing and data loading unit divides the large capacity XML file into blocks of a fixed size so as to enable parallel processing in a plurality of computers, thereby generating a plurality of XML data blocks. Loading the plurality of XML data blocks in the pre-processing and data loading unit into a distributed file system; Obtaining a solution to the linear path patterns in the mapping process of the first mapping task unit using the query index by receiving the divided XML data blocks; Calculating a size of a solution for each linear path pattern in a process of reducing the size of the first mapping task; Distributing branch pattern joining operations among multiple computing nodes in a multi-query optimization module in real time in equal amounts to each other; Transmitting the solutions of the linear paths to be taken as input to the branch pattern join operations assigned to the plurality of nodes, at the execution time in the mapping process of the second mapping task unit; Performing branch pattern join operations assigned to each reducer in the redistribution process of the second mapping task unit with solutions of linear path patterns received in the mapping process and outputting the final result; A method for simultaneous parallel processing of multiple branch pattern queries. Inputting pattern pattern queries and a large capacity XML file from the input unit users; Generating a query index by using linear path patterns of the branch pattern queries input to the input unit by the preprocessing and data loading unit, generating a plurality of XML data blocks from the large capacity XML file, and loading the XML data blocks into the distributed file system; The first mapping task unit receiving the XML data blocks and obtaining a solution to the linear path patterns and calculating a solution size for the linear path patterns; Wherein the second mapping task comprises performing a pattern join operation with the solutions of the linear path patterns and outputting the final result. &Lt; RTI ID = 0.0 &gt; 11. &lt; / RTI &gt; The method of claim 7,
Wherein the first mapping task unit receives the XML data blocks and obtains a solution to linear path patterns and calculates a size of a solution to the linear path patterns,
Creating a query index with linear path patterns commonly existing among a plurality of branch pattern queries and performing filtering processing on the XML elements in parallel with the query index with the query index; Transforming a linear path pattern including // and * into a linear path pattern unique to an input XML document to solve the problem of duplication of the same element due to // / * when creating a query index with a linear path pattern; And collecting the sizes of the linear path solutions for the even workload distribution to the distributed nodes. &Lt; Desc / Clms Page number 19 &gt; The method of claim 7,
Wherein the first mapping task unit receives the XML data blocks and obtains a solution to the linear path patterns and calculates a size of a solution to the linear path patterns, Before performing the step of performing a pattern pattern join operation with the solutions of the path patterns and outputting the final result,
The method according to claim 1, further comprising the step of distributing branch pattern joining operations among the plurality of computing nodes in a uniform manner in real time in the multiple query optimization module. The method of claim 9,
In the multi-query optimization module, distributing the branch pattern joining operations to the plurality of computing nodes in equal amounts in real-
a. Reading correspondence information between a linear path pattern and a branch pattern query;
b. Reading the size information of the linear path solutions; Calculating a cost of a branch pattern join operation using the read linear path pattern mapping information and size information;
c. Arranging the calculated costs in ascending order, and assigning the allocated nodes to an arbitrary redundancy performed node having no allocated pattern pattern query;
d. In the case of the linear path pattern common to the already allocated branch pattern queries, the step of removing from the calculated cost;
e. Repeating the steps c and d until there is no redundancy node having no assigned branch pattern join operation;
f. Searching for a node having the least cost of the allocated branch pattern query;
g. Searching for a branch pattern query having a largest solution size of a linear path pattern common to all the branch pattern nodes assigned by the redundancy performing node found in the step f, and assigning the branch pattern node to the corresponding redundancy performing node;
h. And repeating the steps f and g until no untyped branch pattern query is left. The method of claim 1, The method of claim 9,
Wherein the step of performing a pattern join operation with the solutions of the linear path patterns and outputting the final result comprises:
Referring to the correspondence information between the redistribution node and the branch pattern query allocated by the multiple query optimization module and transmitting the linear path solutions required for each redistribution node; (Key, value) pair transmitted from the map execution unit as the area start value in the area-based numbering technique, and when performing the branching in the redesper performing part, the label value Allowing the user to input and use the joins as the joins; A step of processing a plurality of branch pattern joins by taking the solutions of the transmitted linear path patterns as inputs and outputting a final result, and a step of generating a linear path pattern common to the branch pattern queries, And transmitting the corresponding linear path solution to the corresponding redundancy operation node only once without repeatedly transmitting the linear path solution in the case of allocation.

Images