JP7211255B2 - Search processing program, search processing method and information processing device - Google Patents

Description

The present invention relates to a search processing program, a search processing method, and an information processing apparatus.

In recent years, there has been an increasing interest in collecting, accumulating, and searching data on various kinds of knowledge. Such data may be represented in some graph data model. RDF (Resource Description Framework) is known as one of such graph data models. Currently, there are many open data described by RDF. In the RDF data model, the relationships are expressed using three elements, a subject, a predicate, and an object, which are called triples, as nodes. Data described in RDF is hereinafter referred to as RDF data.

A query language called SPARQL has been standardized for searching and analyzing RDF data. SPARQL is a language similar to SQL, and by using this to describe a query, it is possible to retrieve data that meets conditions from an RDF store that stores RDF data.

As a method for improving the efficiency of such SPARQL query processing, there is a technique called parallelization. As one method of parallelization, there is a method using a distributed framework represented by MapReduce. A method using a distributed framework is, simply put, an approach in which each computer performs parallel processing. In the distributed framework, by increasing the number of computers, it is possible to secure scalability against an increase in the amount of data and to process large-scale data efficiently.

Middleware using such a distributed framework is, for example, Hadoop (registered trademark). Hadoop is middleware that distributes data to multiple servers and processes it in parallel. Because it is capable of high-speed analysis of terabyte-level and even petabyte-level data, it is used as a major technology for big data utilization. there is In Hadoop, one master server cooperates with a large number of slave servers connected under it to perform high-speed data processing. The master server controls the flow of the entire data processing, and the actual calculation processing is performed by subordinate slave servers. Therefore, in Hadoop, the greater the number of slave servers, the higher the processing power, and the more data can be calculated at high speed.

In addition, Hadoop uses two main technologies. One is HDFS (Hadoop Distributed File System). HDFS is a virtual file system capable of collecting hard disks of many slave servers, writing huge amounts of data to be calculated, and writing aggregated results.

Another is MapReduce processing. MapReduce processing is a method of performing calculation processing in two steps: Map processing for extracting and decomposing desired data from given data and Reduce processing for summarizing the extracted data. MapReduce processing is efficient because parallel processing can be performed by a plurality of slave servers. The data distributed on the HDFS is used as the data to be calculated by the MapReduce process.

In addition, there is a conventional technology that extracts variables corresponding to frequently compared values in SPARQL search queries, adds new nodes created by combining values corresponding to the extracted variables to RDF data, and performs search processing. . Also, the triple is stored in a data item in a set of data items ordered according to the data of the triple, and the distributed calculator in which the data item is stored is determined according to the position of the data item in the set. There is prior art. Also, there is a conventional technique for registering a set of a hash value and a key value determined from a character string when the length of the character string exceeds a set threshold when creating an index. There is also a conventional technique for document management using tags and character strings.

WO2014/207827 JP 2013-175181 A JP-A-2000-90115 JP-A-2008-52662

However, for example, RDF data consists of representing the relationship between three elements of subject, predicate and object. On the other hand, when MapReduce processing is performed, input data is treated as a key=value format, that is, as two elements. Therefore, when RDF data is processed by MapRdduce processing, the work of decomposing the RDF data into two-element format of key=value and creating all combinations in advance is added. For example, when the three elements of RDF data are represented as (s, p, o), the combination of (s, p), (p, o) or (o, s) is regarded as one element, and the total is two elements. is decomposed into This conversion work takes an enormous amount of time.

Furthermore, in the case of RDF data, for example, when performing a search in which two of the three elements are determined, the two elements are compared and searched. A long character string, in other words, a value with a large data area can be stored as the value of each element of the RDF data. In particular, it may take an enormous amount of time to search such a long character string as two fixed elements.

Moreover, even if a conventional technique is used in which a new node created by combining values corresponding to frequently appearing variables is added to the RDF data, the difference between the data formats handled by the RDF data and the MapReduce process cannot be resolved, and the search process is difficult to perform at high speed. Also, even if the conventional technique of determining the computer to be arranged according to the position of the data item in the set is used, the difference in data format handled by the RDF data and the MapReduce process is similarly not resolved, and the search process is speeded up. It is difficult to do. In addition, with the conventional technology that registers pairs of hash values and key values based on character strings as indexes, search processing is faster when elements do not differ, but differences in data format when the number of elements differs is also eliminated. Therefore, it is difficult to perform search processing at high speed. Furthermore, even with conventional techniques for document management using tags and character strings, the difference in data format handled between RDF data and MapReduce processing is similarly not eliminated, making it difficult to perform high-speed retrieval processing.

The disclosed technology has been made in view of the above, and aims to provide a search processing program, a search processing method, and an information processing apparatus that perform search processing at high speed.

In one aspect of the search processing program, search processing method, and information processing apparatus disclosed in the present application, a computer is caused to execute the following processes. Two elements are extracted from data having three elements, and a first table is generated in which an identifier smaller in size than the extracted two elements is associated with the extracted two elements. A second table is generated by adding the identifier to the three-element table. The second table is divided and arranged in a plurality of processing units. When retrieving, the first table is used to retrieve the identifier, and the retrieved identifier is used in each of the processing units for a portion of the second table located in each of the processing units. to search. Outputting the rows of the second table extracted by the search.

In one aspect, the present invention can perform search processing at high speed.

FIG. 1 is a system configuration diagram of an information processing system. FIG. 2 is a block diagram showing the details of the master server and slave servers. FIG. 3 is a diagram showing an example of RDF data represented in a tree format. FIG. 4 is a diagram of an example of RDF data represented in tabular form. FIG. 5 is a diagram showing an example of an identifier correspondence table. FIG. 6 is a diagram showing an example of an identifier-attached RDF data table. FIG. 7 is a diagram showing an overview of MapReduce processing. FIG. 8 is a diagram illustrating an overview of MapReduce processing when using parameter identifiers according to the first embodiment. FIG. 9 is a flowchart of processing for generating an identifier table and an identifier-attached RDF data table according to the first embodiment. FIG. 10 is a flowchart of MapReduce processing according to the first embodiment. FIG. 11 is a diagram showing an example of a divided data table. FIG. 12 is a diagram illustrating an overview of MapReduce processing when using parameter identifiers according to the second embodiment. FIG. 13 is a diagram illustrating an example of the hardware configuration of a computer; FIG. 14 is a block diagram showing details of a master server and slave servers according to the third embodiment. FIG. 15 is a diagram showing an example of the identifier correspondence table before division. FIG. 16 is a diagram showing an example of a division identifier correspondence table. FIG. 17 is a diagram illustrating an overview of MapReduce processing when using parameter identifiers according to the third embodiment. FIG. 18 is a flowchart of processing for generating an identifier table and an identifier-attached RDF data table according to the third embodiment. FIG. 19 is a flowchart of MapReduce processing according to the third embodiment.

Exemplary embodiments of a search processing program, a search processing method, and an information processing apparatus disclosed in the present application will be described below in detail with reference to the drawings. The search processing program, search processing method, and information processing apparatus disclosed in the present application are not limited to the following embodiments.

FIG. 1 is a system configuration diagram of an information processing system. The information processing system 1 has a Hadoop cluster 10, an HDFS (Hadoop Distributed File System) client 20, and a job client 30, as shown in FIG.

The HDFS client 20 is an information processing terminal that instructs the Hadoop cluster 10 to manage data. The HDFS client 20 is connected to the master server 11 of the Hadoop cluster 10 via a network. The HDFS client 20 receives an input of a data management instruction from a user from an input device (not shown). Then, the HDFS client 20 transmits to the master server 11 via the HDFS API (Application Programming Interface) a processing instruction for data management according to the input from the user.

The job client 30 is an information processing terminal that issues job management instructions to the Hadoop cluster 10 . The job client 30 is connected to the master server 11 of the Hadoop cluster 10 via a network. The job client 30 has a MapReduce program. The job client 30 receives an input of a job management instruction from a user from an input device (not shown). The job client 30 then transmits to the master server 11 a processing command for job management according to the input from the user.

These HDFS client 20 and job client 30 may be arranged in the same information processing apparatus, or may be arranged in different information processing apparatuses. Also, the functionality of the HDFS client 20 and the job client 30 may be located within the Hadoop cluster 10 .

Hadoop cluster 10 has master server 11 and slave server 12 . Although three slave servers 12 are illustrated in FIG. 1, the number of slave servers 12 is not particularly limited. The master server 11 is connected to each slave server 12 via a network. Furthermore, the master server 11 is connected to the HDFS client 20 and the job client 30 via a network. Each slave server 12 is connected to each other via a network.

FIG. 2 is a block diagram showing the details of the master server and slave servers. The master server 11 and the slave server 12 will be described below with reference to FIG. Here, in FIG. 1, in order to illustrate the outline of the configuration, only the main configuration is described and some configurations are omitted, but the master server 11 and the slave server 12 have the configuration shown in FIG.

The master server 11 has an RDF store 110 , an HDFS 111 , a name node 112 , a metadata DB (Data Base) 113 and a job tracker 114 . Furthermore, the master server 11 has a first generator 115 , a second generator 116 , an RDF controller 117 , a SPARQL processor 118 and a MapReduce processor 119 .

The HDFS 111 is a virtual file system that looks like one file system in cooperation with a plurality of servers. The HDFS 111 performs file management by dividing a file into units called block sizes. The block size is 64MB by default. HDFS 111 does not have an exclusive control function. Also, in the HDFS 111, new creation and addition of files are possible, but modification is not permitted. One Map task is created for one block of HDFS 111 .

RDF data can be represented, for example, in a tree format as shown in FIG. FIG. 3 is a diagram showing an example of RDF data represented in a tree format. The subject is the data surrounded by an ellipse placed at the starting point of the arrow in FIG. Also, the data surrounded by an ellipse arranged at the end point of the arrow corresponds to the predicate. Furthermore, the data written on the arrow corresponds to the predicate.

The RDF data can also be represented in tabular form as shown in FIG. FIG. 4 is a diagram of an example of RDF data represented in tabular form. FIG. 4 is a diagram showing the tree-format RDF data shown in FIG. 3 in tabular form. In FIG. 4, id (Identifier) represents an identifier given to each triple. Also, Subject represents a subject in a triple, predicate represents a predicate, and object represents an object. Corresponding to the identifier assigned to each triple in this way, the subject, predicate and object of the triple are registered corresponding to one row (row) in the tabular format.

The RDF store 110 can hold RDF data in tabular form as shown in FIG. 4, for example. By storing the RDF data stored in the RDF store 110 as an input in the HDFS 111, it becomes possible to manipulate the data in the HDFS 111 by the MapReduce process described later.

Returning to FIG. 2, the description continues. The RDF controller 117 manages RDF data stored in the RDF store 110 . For example, the RDF controller 117 reads or stores designated RDF data in the RDF store 110 in response to a read request or storage request. In addition, the RDF controller 117 receives an instruction to store the RDF data stored in the RDF store 110 in the HDFS 111 , and stores the RFD data stored in the RDF store 110 in the HDFS 111 as an input.

The first generation unit 115 receives an instruction to generate an identifier correspondence table from the HDFS client 20 . The first generator 115 then instructs the RDF controller 117 to acquire the subject, predicate and object of all RDF data registered in the RDF store 110 . After that, the first generator 115 acquires the subjects, predicates and objects of all RDF data registered in the RDF store 110 from the RDF controller 117 .

Next, the first generating unit 115 tabulates the obtained subjects, predicates, and objects by removing duplicates. Then, the first generating unit 115 generates all possible combinations of "subject, predicate", "predicate, object", and "subject, object" using the aggregated subjects, predicates, and objects. These sets are hereinafter referred to as "value patterns". The subject, predicate and object of this RDF data are an example of "three elements", and the two values included in the value pattern are an example of "two elements".

Next, the first generating unit 115 determines whether or not the generated value patterns include a value pattern that is not included in the actual set of subject, predicate and object of each RDF data. If there is a value pattern that is not included in the actual set of subject, predicate and object of each RDF data, the first generation unit 115 generates the value pattern not included in the subject, predicate and object of each RDF data Extract value patterns other than value patterns.

The first generator 115 assigns a pattern identifier to each of the extracted value patterns. This identifier has a smaller data size than the value pattern. The data size is the size that occupies memory. Then, the first generation unit 115 creates an identifier correspondence table representing the correspondence between each value pattern and the assigned pattern identifier. At this time, the first generating unit 115 adds information indicating non-existence of value patterns that are not included in the set of subject, predicate and object of each RDF data and registers them in the identifier correspondence table. . FIG. 5 is a diagram showing an example of an identifier correspondence table.

Here, a case of creating an identifier correspondence table based on the RDF data shown in FIG. 4 will be described. Also, a case of generating a value pattern of a combination of a predicate and an object will be described as an example.

The first generation unit 115 tabulates the predicates 201 in the RDF data shown in FIG. 4 excluding duplicates. In this case, the first generator 115 acquires the two words “likes” and “loves” as predicates. In addition, the first generation unit 115 tabulates the objects 202 in the RDF data shown in FIG. 4 by eliminating duplication. In this case, the first generator 115 acquires four words "A", "C", "D" and "F" as predicates. Then, the first generation unit 115 generates all combinations of the acquired predicates and objects. In this case, the first generation unit 115 generates “likes A”, “likes C”, “likes D”, “likes F”, “loves A”, “loves C”, “loves D” and “loves F”. Generate as a value pattern. Then, the first generating unit 115 determines that "likes A" and "loves D" are not included in the RDF data shown in FIG. After that, the first generation unit 115 assigns pattern identifiers to value patterns that actually exist, associates information indicating non-existence with value patterns that do not actually exist, and creates an identifier correspondence table 211 shown in FIG. and 212. In FIG. 5, an identifier correspondence table 211 representing value patterns including "likes" as a predicate and an identifier correspondence table 212 representing value patterns including "loves" as a predicate are separately described. Furthermore, the first generation unit 115 adds NA (Not Applicable) indicating non-existence to non-existent value patterns “likes A” and “loves D” and registers them in the identifier correspondence tables 211 and 212, respectively.

After that, the first generation unit 115 transmits the generated identifier correspondence table to the RDF controller 117 and instructs the RDF store 110 to store it. Furthermore, the first generation unit 115 notifies the second generation unit 116 of completion of generation of the identifier correspondence table. This identifier correspondence table corresponds to an example of the "first table".

Second generation unit 116 receives from first generation unit 115 notification of the completion of generation of the identifier correspondence table. Second generation unit 116 then transmits an identifier correspondence table acquisition request to RDF controller 117 . After that, the second generator 116 acquires all identifier correspondence tables created by the first generator 115 from the RDF controller 117 .

Next, the second generating unit 116 transmits to the RDF controller 117 an acquisition request for each RDF data registered in the RDF store 110 . Then, the second generation unit 116 confirms the subject, predicate and object of each RDF data acquired from the RDF controller 117, and acquires the pattern identifier corresponding to the value pattern of each combination from the identifier correspondence table. After that, the second generating unit 116 generates an RDF data table with an identifier by adding the acquired pattern identifier for each RDF data to the triple correspondence table of each RDF data.

FIG. 6 is a diagram showing an example of an identifier-attached RDF data table. “vp-id” in FIG. 6 represents a pattern identifier. The pattern identifier 221 corresponds to the value pattern of the combination of subject and predicate. The pattern identifier 222 corresponds to the value pattern of the predicate-object combination. The pattern identifier 223 corresponds to the value pattern of the subject-object combination.

The second generator 116 acquires, for example, "A", "likes" and "D" as the subject, predicate and object of the RDF data on the first line in FIG. Then, the second generation unit 116 acquires value patterns of “A likes”, “likes D”, and “AD” from the acquired values. After that, the second generator 116 acquires a pattern identifier corresponding to the acquired value pattern. For example, the second generation unit 116 acquires the pattern identifier "0002" of "likes D" from the identifier correspondence table 211 of FIG. Similarly, the second generator 116 acquires “2001” and “4001” as pattern identifiers of “A likes” and “AD”. After that, the second generation unit 116 registers each pattern identifier in association with the RDF data in the first row.

After that, the second generating unit 116 transmits the generated identifier-attached RDF data table to the RDF controller 117 and stores it in the RDF store 110 . This identifier-attached RDF data table corresponds to an example of the "second table".

The namenode 112 notifies the RDF controller 117 of a request to acquire the identifier-attached RDF data table stored in the RDF store 110 . The namenode 112 then acquires the identifier-attached RDF data table stored in the RDF store 110 from the RDF controller 117 . Here, when the data of the identifier-attached RDF data table is distributed, the namenode 112 handles the data stored in the RDF store 110 in cooperation with the RDF controller 117. Data may be obtained directly from the RDF store 110 .

The name node 112 determines the data node 121 in which blocks containing partial row data of the identifier-attached RDF data table are stored. Here, although one slave server 12 is shown in FIG. 2 for the sake of clarity, a plurality of slave servers 12 are actually arranged as shown in FIG. , the location of each block is selected from among the data nodes 121 of .

Then, the name node 112 transmits and arranges a block containing part of row data of the identifier-attached RDF data table to the selected data node 121 . Here, the NameNode 112 may transmit multiple blocks to one DataNode 121 . This placement of blocks in each name node 112 corresponds to an example of "dividing and placing in a plurality of processing units". Furthermore, the name node 112 registers information on the data node 121 that is the storage destination of each block in the metadata DB 113 .

Here, in distributed arrangement, the name node 112 duplicates one data block and places it in a plurality of data nodes 121 . For example, NameNode 112 duplicates one data block into three. As a result, when a failure occurs in a certain data node 121, the same block stored in another data node 121 can be used, and the Hadoop cluster 10 is ensured of failure tolerance. This name node 112 corresponds to an example of the "placement section".

The SPARQL processing unit 118 receives SPARQL query input from the job client 30 . Then, the SPARQL processing unit 118 analyzes the obtained SPARQL query and converts it into MapReduce processing. Furthermore, the SPARQL processing unit 118 notifies the RDF controller 117 of an identifier correspondence table acquisition request. After that, the SPARQL processing unit 118 acquires the identifier correspondence table stored in the RDF store 110 from the RDF controller 117 . Next, the SPARQL processing unit 118 refers to the identifier correspondence table and determines whether or not there is a value pattern corresponding to the acquired query element. If there is no value pattern corresponding to the acquired query element, the SPARQL processing unit 118 returns the search result to the job client 30 as zero matching results for such a value pattern.

On the other hand, if there is a value pattern corresponding to the acquired query element, the SPARQL processing unit 118 acquires the pattern identifier assigned to the value pattern of the acquired query element. Next, the SPARQL processing unit 118 replaces the character string with the obtained pattern identifier in the MAPReduce process. After that, the SPARQL processing unit 118 outputs MapReduce processing including the pattern identifier to the MapReduce processing unit 119 .

After that, the SPARQL processing unit 118 receives the input of the execution result of the MapReduce processing from the MapReduce processing unit 119 . Then, the SPARQL processing unit 118 transmits the acquired execution result of the MapReduce process to the job client 30 as the execution result of the SPARQL query. The SPARQL processing unit 118 corresponds to an example of the "output unit".

The MapReduce processing unit 119 receives a MapReduce processing input including a pattern identifier from the SPARQL processing unit 118 . This MapReduce process includes multiple MapReduce processes corresponding to individual search processes included in the original SPARQL process. Therefore, the MapReduce processing unit 119 acquires each MapReduce process included in the received MapReduce process. The MapReduce processing unit 119 then instructs the job tracker 114 to execute each acquired MapReduce process. In this case, since the character string used for the search is replaced with the pattern identifier, the MapReduce processing unit 119 instructs the job tracker 114 to execute the MapReduce process using the pattern identifier.

After that, the MapReduce processing unit 119 receives an input of the execution result of the MapReduce processing from the job tracker 114 . The MapReduce processing unit 119 then outputs the execution result of the MapReduce processing to the SPARQL processing unit 118 .

The job tracker 114 receives an instruction to execute each MapReduce process from the MapReduce processing unit 119 . Next, the job tracker 114 confirms the data node 121 where each block stored in the metadata DB 113 is arranged, and determines the data node 121 to execute each MapReduce process. The job tracker 114 then generates and assigns one Map task to one block. Job tracker 114 then sends each Map task to task tracker 123 of slave server 12 having a data node 121 holding the corresponding block. In this way, the communication cost can be minimized by allocating each Map task to the slave server 12 having a block targeted by each Map task.

After that, the job tracker 114 receives the job execution result from the task tracker 123 of each slave server 12 . Then, the job tracker 114 outputs to the MapReduce processing unit 119 an execution result of the MapReduce process, which summarizes the job execution results.

Next, the slave server 12 will be explained. The slave server 12 has a data node 121, an HDFS 122, a task tracker 123 and a MapReduce processing unit 124, as shown in FIG.

The HDFS 122 manages data in block units with a default size of 64 MB, like the HDFS 111 . Each HDFS 122 of each slave server 12 forms one virtual file system as a whole.

The data node 121 receives from the name node 112 a block containing partial row data of the RDF data table with identifier. Here, data node 121 may receive multiple blocks. Then, the data node 121 stores the obtained block in the HDFS 122 of its own device. That is, the HDFS 122 stores the data of some of the rows of the RDF data table with identifiers.

The task tracker 123 receives from the job tracker 114 the Map task corresponding to the blocks it owns. The task tracker 123 then instructs the MapReduce processing unit 124 to execute the Map processing instructed by the Map task. After that, the task tracker 123 receives the input of the execution result of the MapReduce process executed according to the MapTask from the MapReduce processing unit 124 . The task tracker 123 then transmits the execution result of each Map task to the job tracker 114 .

The MapReduce processing unit 124 executes MapReduce processing according to the Map Task acquired from the task tracker 123 . Here, the MapReduce process will be described with reference to FIG. FIG. 7 is a diagram showing an overview of MapReduce processing. Here, a case where the MapReduce processing units 124A to 124C operate will be described. Further, here, the MapReduce processing unit 124A processes blocks 301 to 303, and the MapReduce processing units 124B and 124C process other blocks. Data of each block includes data having a format of key=value. Of the two characters enclosed in parentheses in FIG. 7, the first character represents key and the second character represents value. Furthermore, here, a case where a process of counting data whose value is X is executed as a MapReduce process will be described.

The MapReduce processing unit 124A receives the data of blocks 301 to 303 as input, gives the input to the map function, and outputs the result of internal processing as new data in the key=value format. Here, the MapReduce processing unit 124A outputs data whose Value is X. In this case, the MapReduce processing unit 124A extracts (K1, X) and (K4, X) from the block 301, extracts (K2, X) and (K3, X) from the block 302, and extracts (K2 , X) and (K5, X). This processing corresponds to the Map processing. Map processing is performed for each block 301-303.

Similarly, the MapReduce processing unit 124B extracts blocks whose value is X from the blocks to be processed. In this case, the MapReduce processing unit 124B extracts (K1, X), (K4, X), (K5, X), (K1, X) and (K6, X). Similarly, the MapReduce processing unit 124C also extracts blocks whose value is X from the blocks to be processed.

Next, the MapReduce processing units 124A to 124C classify each extracted data and transmit each of them to a predetermined transmission destination among the MapReduce processing units 124A to 124C. For example, in FIG. 7, the data with keys K1 and K2 are combined into the MapReduce processing unit 124A. Also, data with keys K3 and K4 are put together in the MapReduce processing unit 124B. Also, data with keys K5 and K6 are put together in the MapReduce processing unit 124C. Next, each of the MapReduce processing units 124A to 124C rearranges the collected data. Here, the MapReduce processing units 124A to 124C rearrange the data so that they are grouped for each key. In other words, the MapReduce processing units 124A to 124C aggregate the key=value format data having the same key. These processes are called shuffle and sort processes.

Next, the MapReduce processing units 124A to 124C obtain the data for which the shuffle and sort processing have been completed, process the obtained data inside the Reduce function, and output the result as data in the key=value format. Here, the MapReduce processing units 124A to 124C aggregate each data having the same key as the Reduce function. In FIG. 7, the MapReduce processing unit 124A outputs (K1, 3) as data representing that there are three (K1, X). Also, the MapReduce processing unit 124A outputs (K2, 2) as data indicating that there are two (K2, X). The MapReduce processing unit 124B outputs the totalization result of the data whose key is K3 or K4. The MapReduce processing unit 124C outputs the totalization result of the data whose key is K5 or K6. This processing is called Reduce processing. The Reduce process can be edited by the user. Here, as the Reduce process, a process of aggregating data having the same key is performed, but it is also possible to change to other processes. For example, when returning a result corresponding to a SPARQL query, the Reduce process may be a process of outputting data whose value is X and whose key is the value corresponding to X as it is.

The MapReduce processing unit 124 has a Map task execution unit 241 and a Reduce task execution unit 242 . The Map task execution unit 241 performs Map processing and shuffle and sort processing.

The Map task execution unit 241 acquires a Map task whose execution has been instructed by the task tracker 123 . Then, the Map task execution unit 241 executes Map processing. In this case, the Map task execution unit 241 receives the Map task using the parameter identifier. Therefore, the Map task execution unit 241 executes Map processing using parameter identifiers as shown in FIG. 8, for example. FIG. 8 is a diagram illustrating an overview of MapReduce processing when using parameter identifiers according to the first embodiment. A case where the RDF data tables 411 and 421 with identifiers shown in FIG. 8 are processed by different MapReduce processing units 124 will be described.

For example, in FIG. 8, the Map task execution unit 241 acquires a Map task that extracts data whose value is the parameter identifier 1002 surrounded by a thick frame. Here, in FIG. 8, the value pattern corresponding to 1002 is described for easy understanding, but the actual Map task may not include the value pattern.

The Map task execution unit 241 acquires data to be subjected to Map processing from the identifier-attached RDF data table 411 or 421, converts the data into key=value format data, and receives the data as input. Here, the Map task execution unit 241 receives data with a subject and a key and a value pattern of a combination of a predicate and an object as the value.

Then, each Map task execution unit 241 extracts data 412 or 422 whose pattern identifier representing value is 1002 from the input data. Each Map task execution unit 241 then shuffles and sorts the extracted data 412 or 422 . Here, each Map task execution unit 241 collects data whose key is B to one side, and collects other data to the other side. The process of collecting data in each slave server 12 for each key corresponds to an example of the process of "summarizing based on any one of the three elements".

Furthermore, each Map task execution unit 241 sorts the collected data based on the key to generate data 413 or 423 . Each Map task execution unit 241 then outputs the data 413 or 423 to the Reduce task execution unit 242 . This Map task execution unit 241 corresponds to an example of a "search unit".

The Reduce task execution unit 242 performs Reduce processing. For example, as shown in FIG. 8, each Reduce task execution unit 242 receives input of data 413 or 423 from the corresponding Map task execution unit 241 respectively. Next, each Reduce task execution unit 242 tallies the number of data having the same key from the acquired data 413 or 423 . Each Reduce task execution unit 242 then outputs a result 414 or 424 of the Reduce process to the task tracker 123 . Here, c in results 414 and 424 in FIG. 8 represents the count value. Aggregation of the number of data having the same key by the Reduce task execution unit 242 corresponds to an example of “execution of predetermined processing”.

Next, with reference to FIG. 9, a flow of processing for generating an identifier table and an identifier-added RDF data table according to the first embodiment will be described. FIG. 9 is a flowchart of processing for generating an identifier table and an identifier-attached RDF data table according to the first embodiment. In the following, the intermediate operation of the RDF controller 117 in data transmission/reception with the HDFS 111 will be omitted.

The first generating unit 115 obtains the RDF data stored in the RDF store 110 by excluding duplication of all subjects, predicates and objects. Then, the first generation unit 115 combines two each of the acquired subject, predicate and object to extract a value pattern (step S1).

Next, the first generator 115 determines whether or not the extracted value patterns include a value pattern that does not exist in the actual RDF data stored in the RDF store 110 (step S2). If there is no value pattern that does not actually exist (step S2: No), the first generator 115 proceeds to step S4.

If there is a value pattern that does not actually exist (step S2: Yes), the first generation unit 115 extracts the value pattern that actually exists by removing the value pattern that does not actually exist from the extracted value patterns. (step S3).

Next, the first generating unit 115 assigns identifiers to actually existing value patterns, and generates an identifier correspondence table representing pattern identifiers corresponding to each value pattern (step S4). After that, the first generation unit 115 causes the RDF controller 117 to store the generated identifier correspondence table in the RDF store 110, and notifies the second generation unit 116 of the completion of the generation of the identifier correspondence table.

The second generating unit 116 that has received the notification of the completion of generation of the identifier pair table acquires from the RDF store 110 all the RDF data and the identifier correspondence table contained in the RDF store 110 . Next, the second generation unit 116 acquires a value pattern that combines the subject and predicate of each RDF data, a value pattern that combines the predicate and object, and a value pattern that combines the subject and object. . Then, the second generating unit 116 acquires pattern identifiers corresponding to the acquired value patterns from the identifier correspondence table. Next, the second generating unit 116 generates an RDF data table with an identifier by adding the acquired pattern identifier to each piece of RDF data in the correspondence table representing triple correspondence (step S5). After that, the second generation unit 116 stores the generated identifier-attached RDF data table in the RDF store 110 . Here, in the present embodiment, the second generation unit 116, which receives the notification from the first generation unit 115, automatically generates RDF data with an identifier, but other procedures may be used. . For example, the second generation unit 116 may receive an instruction from a user using the job client 30 and use the input of the instruction as a trigger to generate identifier-attached RDF data.

The NameNode 112 acquires the RDF data table with identifier from the RDF store 110 . Next, the namenode 112 determines the datanode 121 in which each block containing the data registered in the identifier-attached RDF data table is placed. Then, the namenode 112 transmits each block including a part of row data of the identifier-attached RDF data table to each of the datanodes 121 determined as allocation destinations, and executes data distribution allocation (step S6).

Next, the flow of MapReduce processing according to the first embodiment will be described with reference to FIG. FIG. 10 is a flowchart of MapReduce processing according to the first embodiment.

The SPARQL processing unit 118 receives an input of a SPARQL query execution command from the job client 30 . The SPARQL processing unit 118 then executes the SPARQL query (step S11).

Next, the SPARQL processing unit 118 converts the SPARQL query into a job for MapReduce processing (step S12).

Next, the SPARQL processing unit 118 acquires the identifier correspondence table from the HDFS 111. FIG. Then, the SPARQL processing unit 118 parses the entered query and determines whether there is a value pattern corresponding to the value patterns registered in the identifier correspondence table (step S13). If there is no corresponding value pattern (step S13: No), the SPARQL processing unit 118 returns to the job client 30 a search result indicating that the matching result for such a value pattern is 0, and ends the SPARQL query execution processing. do. In practice, the SPARQL processing unit 118 can know whether or not there is a value pattern corresponding to the value pattern registered in the identifier correspondence table at the parsing stage.

On the other hand, if there is a corresponding value pattern (step S13: Yes), the SPARQL processing unit 118 refers to the pattern identifier and executes the MapReduce process.

The MapReduce processing unit 119 receives an instruction from the SPARQL processing unit 118, refers to the pattern identifier, and instructs the job tracker 114 to execute the MapReduce process. The job tracker 114 confirms the metadata DB 113 and selects a slave server 12 to perform the MapReduce process. The job tracker 114 then divides the MapReduce process into block-based Map tasks and transmits them to the selected slave server 12 . The task tracker 123 receives Map tasks from the job tracker 114 . The task tracker 123 then instructs the MapReduce processing unit 124 to execute the acquired Map task. The MapReduce processing unit 124 receives an instruction to execute a Map task from the task tracker 123 . The Map task execution unit 241 then uses the RDF data with identification code stored in the HDFS 122 to execute the Map process designated by the Map task (step S14).

Next, the Map task execution unit 241 shuffles the processing results of the Map processing so that they are collected for each key and distributes them to the Map task execution unit 241 of each slave server 12 . Further, the Map task execution unit 241 sorts the data distributed to its own device by shuffling so that the data are grouped for each key (step S15). The Map task execution unit 241 then outputs the sorted data to the Reduce task execution unit 242 .

The Reduce task execution unit 242 executes a predesignated Reduce process on the data acquired from the Map task execution unit 241 (step S16). For example, the Reduce task execution unit 242 aggregates data for each key.

After that, the Reduce task execution unit 242 outputs the result of the MapReduce process to the task tracker 123 . The task tracker 123 transmits the input result of the MapReduce process to the job tracker 114 of the master server 11 . The job tracker 114 collects the MapReduce processing results sent from each slave server 12 . The job tracker 114 then combines the results of the MapReduce processing. The job tracker 114 then transmits the combined result of the MapReduce processing to the SPARQL processing unit 118 via the MapReduce processing unit 119 . The SPARQL processing unit 118 receives the results of the combined MapReduce processing, and converts the received data into RDF format (step S17). After that, the SPARQL processing unit 118 transmits the result of the MapReduce process converted into the RDF format to the job client 30 as the execution result of the SPARQL query.

As described above, the Hadoop cluster according to the present embodiment assigns identifiers to value patterns that are combinations of two elements out of three elements included in graph data, and uses the identifiers to execute MapReduce processing. . As a result, when searching for graph data in the MapReduce process, the search can be performed using an identifier with a small data area, and matching at the time of search can be performed at high speed.

Furthermore, the Hadoop cluster according to this embodiment creates an identifier correspondence table by excluding value patterns that do not exist in actual RDF data. As a result, processing using non-existent RDF data can be omitted, further improving search speed. For example, when receiving an instruction for a search operation using a value pattern that does not exist in RDF data, the Hadoop cluster according to this embodiment can return a result without performing MapReduce processing.

Next, Example 2 will be described. The Hadoop cluster according to the present embodiment differs from the first embodiment in that a divided data table in which value patterns and corresponding keys are registered is used as data to be searched. A Hadoop cluster 10 according to this embodiment is also represented in FIGS. In the following description, descriptions of the functions of the same units as in the first embodiment will be omitted.

The second generator 116 acquires all R RDF data and the identification correspondence table from the RDF store 110 via the RDF controller 117 . Next, the second generation unit 116 acquires a combined value of two of the subject, predicate, and object of each RDF data, and acquires an identifier corresponding to the value pattern that matches the combined value from the identifier correspondence table. do. Then, for each combination of two of the subject, predicate, and object, the second generation unit 116 generates a pattern identifier corresponding to the value pattern, is associated with the remaining one element of to generate a split data table.

FIG. 11 is a diagram showing an example of a divided data table. As shown in FIG. 11, the second generation unit 116 according to the present embodiment generates divided data tables 501 to 503 that match the pattern identification information and the key for each data that has multiple matching key types.

Specifically, the second generating unit 116 generates the divided data table 501 representing the correspondence between the pattern identifier corresponding to the value pattern representing the combination of the predicate and the object and the subject. The second generating unit 116 also generates a divided data table 502 representing the correspondence between the pattern identifier corresponding to the value pattern representing the combination of the subject and the predicate and the object. Also, the divided data table 502 generates a divided data table 503 representing the correspondence between the pattern identifier corresponding to the value pattern representing the combination of the subject and the object and the predicate. Then, the second generating unit 116 stores the generated divided data tables 501 to 503 in the RDF store 110 via the RDF controller 117 .

The namenode 112 transmits a block containing partial row data of the partitioned data table to each datanode 121 . The data node 121 stores in the HDFS 122 blocks containing partial row data of the partitioned data table.

The Map task execution unit 241 receives a Map task execution instruction from the task tracker 123 . The Map task execution unit 241 then selects a table to be used in the Map task. For example, if the Map task uses a value pattern that combines a predicate and an object, the Map task execution unit 241 selects a divided data table in which a value pattern that combines a predicate and an object is registered. Taking the case of FIG. 11 as an example, for example, in the case of Map processing using value patterns of combinations of subjects and predicates, the Map task execution unit 241 selects the divided data table 501 .

Then, the Map task executing unit 241 executes the Map task instructed to execute by the task tracker 123 for each block stored in the HDFS 122 . After that, the Map task execution unit 241 outputs the result of executing the Map processing and the shuffle and sort processing to the Reduce task execution unit 242 .

Now, with reference to FIG. 12, the flow of Map processing by the Map task execution unit 241 according to the second embodiment will be described. FIG. 12 is a diagram illustrating an overview of MapReduce processing when using parameter identifiers according to the second embodiment. Here, a case where the divided data tables 511 and 521 shown in FIG. 12 are processed by different MapReduce processing units 124 will be described.

For example, in FIG. 12, the Map task execution unit 241 acquires a Map task that extracts the parameter identifier 1002 surrounded by a thick frame as a value. Next, the Map task execution unit 241 acquires data to be subjected to Map processing from the divided data table 511 or 521 . In this case, since the data of the divided data tables 511 and 521 are already in the format of key=value, each Map task execution unit 241 can directly input the data of the divided data tables 511 and 521 .

Then, each Map task execution unit 241 extracts data 512 or 522 whose pattern identifier is 1002 and corresponds to value from the input data. Next, each Map task execution unit 241 performs shuffle and sort processing on the extracted data 512 or 522 to acquire data 513 and 523 .

The Reduce task execution unit 242 acquires the results of Map processing and shuffle and sort processing from the Map task execution unit 241 . Then, the Reduce task execution unit 242 performs Reduce processing on the acquired data.

Here, a flow of Reduce processing by the Reduce task execution unit 242 according to the second embodiment will be described with reference to FIG. 12 . Each Reduce task execution unit 242 receives input of data 513 or 523 from the corresponding Map task execution unit 241 . Next, each Reduce task execution unit 242 tallies the number of data having the same key from the acquired data 513 or 523 . Then, each Reduce task execution unit 242 outputs the result 514 or 524 of the Reduce process to the task tracker 123 .

As described above, the Hadoop cluster according to the present embodiment executes MapReduce processing using a partitioned data table representing correspondence between identifiers corresponding to value patterns and keys corresponding to the identifiers. The Hadoop cluster according to this embodiment selects a partitioned data table according to the purpose of Map processing. Since each divided data table is smaller in size than the identifier-added RDF data table used in the first embodiment, the memory consumption can be suppressed and the table can be scanned quickly compared to the first embodiment. can.

Here, in each of the above embodiments, a Hadoop cluster was used for explanation, but the system configuration is not limited to this, and other systems may be used as long as the system uses data having three elements for processing two elements. system configuration. Further, although the RDF data is used in the above embodiments, the same processing can be performed using other data as long as it is graph data, and the same effects can be obtained.

(Hardware configuration)
The master server 11 and slave server 12 according to each of the embodiments described above can be realized by a computer having a hardware configuration as shown in FIG. 13, for example. FIG. 13 is a diagram illustrating an example of the hardware configuration of a computer; The computer 90 has a CPU (Central Processing Unit) 91 , RAM (Random Access Memory) 92 , ROM (Read Only Memory) 93 and HDD (Hard Disk Drive) 94 . Further, the computer 90 has a communication interface (I/F) 95 , an input/output interface (I/F) 96 and a media interface (I/F) 97 .

CPU91 operates based on the program stored in ROM93 or HDD94, and controls each part. The ROM 93 stores a boot program executed by the CPU 91 when the computer 90 is started, a program depending on the hardware of the computer 90, and the like.

The HDD 94 stores programs executed by the CPU 91 and data used by these programs. The communication interface 95 receives data from other devices via the network, sends the data to the CPU 91, and transmits data generated by the CPU 91 to the other devices via the network.

The CPU 91 controls output devices such as a display and a printer and input devices such as a keyboard and a mouse through an input/output interface 96 . The CPU 91 acquires data from the input device via the input/output interface 96 . The CPU 91 also outputs the generated data to the output device via the input/output interface 96 .

The media interface 97 reads programs or data stored in the recording medium 98 and provides them to the CPU 91 via the RAM 92 . The CPU 91 loads the program from the recording medium 98 onto the RAM 92 via the media interface 97 and executes the loaded program. The recording medium 98 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, the RAM 92 and HDD 94 of the computer 90 implement the functions of HDFS 111 and 122 and metadata DB 113 . Furthermore, the CPU 91 of the computer 90 implements the functions of the name node 112, the job tracker 114, the first generator 115, and the second generator 116 by executing the programs loaded on the RAM 92. FIG. Also, the CPU 91 of the computer 90 implements the functions of the RDF controller 117 , the SPARQL processing unit 118 and the MapReduce processing unit 119 . Furthermore, the CPU 91 of the computer 90 implements the functions of the data node 121 , task tracker 123 and MapReduce processing unit 124 .

The CPU 91 of the computer 90 reads these programs from the HDD 94 and executes them, but as another example, the programs may be read from the recording medium 98 or obtained from another device via a network. good.

Here, in the above description, the case where the SPARQL processing unit 118 acquires the pattern identifier corresponding to the value pattern to be searched specified in the SPARQL query from the identifier correspondence table has been described. It is also possible to run on the side. For example, the MapReduce processing unit 124 of the slave server 12 may acquire the pattern identifier corresponding to the value pattern to be searched from the identifier correspondence table and execute the MapReduce processing using the acquired pattern identifier.

Next, Example 3 will be described. In MapReduce processing by Hadoop, both input data and final output data are stored in HDFS. Furthermore, in MapReduce processing by Hadoop, an intermediate file generated in Map processing is also temporarily stored in HDFS. Therefore, intermediate files are input/output to/from HDFS in Map processing. HDFS is a file system arranged in HDDs and SSDs (Solid State Drives), and the time required for reading and writing is longer than for arithmetic processing. Therefore, when performing MapReduce processing by Hadoop, there is a possibility that a delay may occur.

Therefore, a method of reducing access to HDFS and improving processing speed by using in-memory processing that performs processing using data in memory when performing MapReduce processing is conceivable. For example, as distributed in-memory processing, there is processing using Spark (registered trademark). MapReduce can be performed in-memory by using Spark.

Spark uses Hadoop HDFS as storage. Therefore, even when Spark is used, input data and final output data are stored in HDFS. On the other hand, intermediate data in map processing is held in memory in RDD (Resilient Distributed Dataset) format, and is processed continuously without being stored in HDFS. Therefore, in the case of repeating Map processing using processing results in deep learning or the like, it is possible to further improve the processing speed compared to MapReduce processing by Hadoop.

However, when data processing is completed in the main memory using distributed in-memory processing such as Spark, in the configuration where the identifier correspondence table is developed in memory, if the size of the identifier correspondence table is large, it is developed in memory. is difficult. In that case, there is a possibility that the processing speed will decrease because the memory capacity allocated to the Map processing in the memory will be insufficient.

Therefore, in the information processing system according to the present embodiment, the identifier correspondence table to be expanded on the memory is reduced by dividing the identifier correspondence table. The following mainly describes the use of the divided identifier correspondence table in the MapReduce process using Spark. FIG. 14 is a block diagram showing details of a master server and slave servers according to the third embodiment. In the following description, the description of the operation of each unit similar to that of the first embodiment will be omitted.

As shown in FIG. 14, the master server 11 has a Spark processing unit 131 in addition to each unit of the first embodiment. Also, the slave server 12 has an SSD 125 and a memory 126 in addition to the components of the first embodiment. Furthermore, the MapReduce processing unit 124 of the slave server 12 according to this embodiment has a memory management unit 243 in addition to the Map task execution unit 241 and the Reduce task execution unit 242 .

The first generation unit 115 receives an instruction to generate an identifier correspondence table from the HDFS client 20 . The first generator 115 then instructs the RDF controller 117 to acquire the subject, predicate and object of all RDF data registered in the RDF store 110 . After that, the first generator 115 acquires the subjects, predicates and objects of all RDF data registered in the RDF store 110 from the RDF controller 117 .

Next, the first generating unit 115 tabulates the obtained subjects, predicates, and objects by removing duplicates. Then, the first generation unit 115 generates value patterns of all possible combinations using the aggregated subjects, predicates, and objects. Next, the first generation unit 115 extracts value patterns other than value patterns that are not included in the actual combinations of subject, predicate and object of each RDF data, and assigns identifiers to them. Then, the first generating unit 115 adds information indicating the absence of value patterns that are not included in the set of the subject, predicate, and object of each RDF data, and assigns them to each value pattern. Create an identifier correspondence table representing the correspondence with pattern identifiers.

At this stage, first generation unit 115 generates identifier correspondence table 213 shown in FIG. FIG. 15 is a diagram showing an example of the identifier correspondence table before division. This identifier correspondence table 213 includes an area 214 representing a value pattern combining a predicate and an object, an area 215 representing a value pattern combining a subject and a predicate, and an area representing a value pattern combining a subject and an object. 216 are included.

Here, for example, in a SPARQL query such as "select ?s where {?s likes C}", a value pattern combining a predicate and an object is retrieved. That is, in this SPARQL query, the areas 215 and 216 in the identifier correspondence table 213 do not have to be searched. Thus, depending on whether the search is a subject-based search that finds the corresponding subject, an object-based search that finds the corresponding object, or a predicate-based search that finds the corresponding predicate, , the areas actually required in the identifier correspondence table 213 are different.

Then, the first generating unit 115 divides the identifier correspondence table 213 into a divided identifier correspondence table 231 for subject-based retrieval, a divided identifier correspondence table 232 for object-based retrieval, and a predicate-based A split identifier correspondence table 233 for retrieval is generated. FIG. 16 is a diagram showing an example of a division identifier correspondence table.

After that, the first generation unit 115 transmits the generated split identifier correspondence tables 231 to 233 to the RDF controller 117 and instructs the RDF store 110 to store them. Furthermore, the first generation unit 115 notifies the second generation unit 116 of completion of generation of the identifier correspondence table. As a result, the division identifier correspondence tables 231 to 233 are stored in the RDF store 110 by the RDF controller 117 .

The SPARQL processing unit 118 receives SPARQL query input from the job client 30 . Then, the SPARQL processing unit 118 analyzes the obtained SPARQL query and converts it into MapReduce processing. After that, the SPARQL processing unit 118 outputs MapReduce processing including the pattern identifier to the Spark processing unit 131 . Furthermore, the SPARQL processing unit 118 notifies the name node 112 of a transmission request for the division identifier correspondence tables 231 to 233. FIG.

The name node 112 receives from the SPARQL processing unit 118 notification of the transmission request for the split identifier correspondence tables 231 to 233 . The name node 112 then acquires the split identifier correspondence tables 231 to 233 from the RDF store 110 and transmits them to the data node 121 . Also, the name node 112 transmits and arranges a block including a part of row data of the RDF data table with identifier to the selected data node 121 .

The Spark processing unit 131 receives from the SPARQL processing unit 118 an input for MapReduce processing to be executed using Spark. Next, the Spark processing unit 131 acquires each MapReduce process included in the received MapReduce process. Then, the Spark processing unit 131 instructs the job tracker 114 to execute the acquired MapReduce process. Furthermore, when MapReduce processing is repeatedly performed using execution results in deep learning or the like, the Spark processing unit 131 manages the repetition procedure and instructs the job tracker 114 to repeatedly execute the MapReduce processing on the memory 126. instruct.

After that, the Spark processing unit 131 receives an input of the execution result of the MapReduce process from the job tracker 114 . The Spark processing unit 131 then outputs the execution result of the MapReduce processing to the SPARQL processing unit 118 . The Spark processing unit 131 in this case corresponds to "Driver" in Spark.

The data node 121 receives from the name node 112 a block containing partial row data of the RDF data table with identifier. Here, data node 121 may receive multiple blocks. Then, the data node 121 stores the obtained block in the HDFS 122 of its own device.

The data node 121 also receives the division identifier correspondence tables 231 to 233 from the name node 112 . Then, the data node 121 stores the acquired split identifier correspondence tables 231 to 233 in the HDFS 122 of the next device.

The MapReduce processing unit 124 according to this embodiment has a Map task execution unit 241 , a Reduce task execution unit 242 and a memory management unit 243 . The MapReduce processing unit 124 acquires a Map task from the task tracker 123 and executes MapReduce processing using Spark. The MapReduce processing unit 124 in this case corresponds to "Exector" in Spark. The details of the MapReduce process using Spark will be described below.

The memory management unit 243 acquires the value pattern to be searched specified in the Map task acquired from the task tracker 123 . Then, the memory management unit 243 specifies whether the search by the value pattern corresponds to a subject-based search, an object-based search, or a predicate-based search. Then, the memory management unit 243 acquires from the SSD 125 a table corresponding to the identified type of search among the division identifier correspondence tables 231 to 233 . Here, a case where subject-based retrieval is performed will be described. That is, the memory management unit 243 acquires the split identifier correspondence table 231 for subject-based retrieval from the SSD 125 . Then, the memory management unit 243 converts the acquired division identifier correspondence table 231 into an RDD. After that, the memory management unit 243 develops the split identifier correspondence table 231 converted into the RDD on the memory 126 .

Also, the memory management unit 243 acquires a block containing part of row data of the identifier-attached RDF data table stored in the HDFS 122 . The memory management unit 243 then converts the acquired block into an RDD. After that, the memory management unit 243 develops the blocks converted into RDDs on the memory 126 . An RDD is a partitioned collection that is immutable and can be executed in parallel. An RDD can be held in memory and has features such as fault tolerance and data locality.

After that, when the memory management unit 243 receives notification of the completion of the Reduce process from the Reduce task execution unit 242 , it acquires the execution result of the MapReduce process from the memory 126 . Then, the memory management unit 243 converts the acquired execution result of the MapReduce process from the RDD format to the data format for storage in the HDFS 111 and stores it in the HDFS 122 . That is, the HDFS 122 stores an identifier-attached RDF data table in which data used in the MapReduce process is stored, and the execution result of the MapReduce process.

The Map task execution unit 241 executes Map processing in a Map task using Spark that has received an execution instruction from the task tracker 123 . Specifically, the Map task execution unit 241 acquires the value pattern to be searched specified by the Map task. Then, the Map task execution unit 241 searches the divided identifier correspondence table 231 on the memory 126 with the acquired value pattern to acquire the parameter identifier corresponding to the value pattern.

Next, the Map task execution unit 241 acquires data to be subjected to Map processing from the identifier-attached RDF data table, converts the data into key=value format data, and inputs the data. Next, the Map task execution unit 241 extracts data that matches the pattern identifier obtained by value from the input data converted to RDD. Next, the Map task execution unit 241 shuffles and sorts the extracted data. Each Map task execution unit 241 then outputs the shuffled and sorted data to the Reduce task execution unit 242 .

Here, the Map task execution unit 241 continuously executes the above processes while holding intermediate data generated during the above processes in the memory 126 in RDD format. In particular, when performing processing by repeatedly using the execution results of the MapReduce processing in deep learning or the like, the Map task execution unit 241 can continuously and repeatedly perform processing by reading data from and writing data to the memory 126 .

The Reduce task execution unit 242 performs Reduce processing. The Reduce process can execute a process predetermined by the Reduce designer. For example, the Reduce task execution unit 242 performs processing such as totaling and aggregating values. After that, the Reduce task execution unit 242 stores the execution result of the MapReduce process in the memory 126 . Furthermore, the Reduce task execution unit 242 notifies the memory management unit 243 and the task tracker 123 of the completion of the Reduce process.

The task tracker 123 receives notification of the completion of the Reduce process from the Reduce task execution unit 242 . The task tracker 123 acquires the execution result of the MapReduce process from the HDFS 122 and transmits it to the job tracker 114 .

Here, in the above description, the partition identifier correspondence tables 231 to 233 are stored in the SSD 125 held by the slave server 12, but there is no particular restriction on the location of the partition identifier correspondence tables 231 to 233. For example, the split identifier correspondence tables 231 to 233 may be arranged in the master server 11 and the memory management unit 243 of the slave server 12 may obtain the split identifier correspondence tables 231 to 233 from the master server 11 .

Next, with reference to FIG. 17, an overview of the MapReduce process when using the parameter identifier according to the third embodiment will be described. FIG. 17 is a diagram illustrating an overview of MapReduce processing according to the third embodiment. Here, a case where the RDF data tables 611 and 621 with identifiers shown in FIG. 17 are processed by different MapReduce processing units 124 will be described.

For example, in FIG. 17, the Map task execution unit 241 acquires a Map task that uses Spark to extract data whose SPARQL query is represented by the syntax “select ?s where {?s loves C.}. Memory management The unit 243 converts the RDF data tables 611 and 621 with identifiers and the divided identifier correspondence table 231 into RDDs and stores them in the memory 126 .

The Map task execution unit 241 acquires 1002 as a pattern identifier corresponding to "loves C" from the divided identifier correspondence table 231 converted into RDD and stored on the memory. Then, the map task execution unit 241 acquires data to be subjected to map processing from the identifier-added RDF data table 411 or 421 converted to RDD, converts it to data in the key=value format, and uses the data as input. . Here, the Map task execution unit 241 receives data with a subject and a key and a value pattern of a combination of a predicate and an object as the value.

Then, each Map task execution unit 241 extracts data 612 or 622 whose pattern identifier representing value is 1002 from the input data and stores it in the memory 126 . Each Map task executing unit 241 executes shuffle and sort processing on the extracted data 612 or 622 and stores the processing result in the memory 126 . Here, each Map task execution unit 241 collects data whose key is B to one side, and collects other data to the other side. Furthermore, each Map task execution unit 241 sorts the collected data based on the key to generate data 613 or 623 and stores the data 613 or 623 in the memory 126 .

The Reduce task execution unit 242 acquires input of data 613 or 623 from the memory 126 . Next, the Reduce task execution unit 242 tallies the number of data having the same key from the acquired data 613 or 623 . Then, the Reduce task execution unit 242 stores the result 614 or 624 of the Reduce process on the memory 126 . Here, c in results 614 and 624 in FIG. 17 represents the count value.

Next, with reference to FIG. 18, the flow of processing for generating an identifier table and an identifier-attached RDF data table according to the third embodiment will be described. FIG. 18 is a flowchart of processing for generating an identifier table and an identifier-attached RDF data table according to the third embodiment. In the following, the intermediate operation of the RDF controller 117 in data transmission/reception with the HDFS 111 will be omitted.

The first generating unit 115 obtains the RDF data stored in the RDF store 110 by excluding duplication of all subjects, predicates and objects. Then, the first generation unit 115 combines two each of the obtained subjects, predicates, and objects to extract a value pattern (step S101).

Next, the first generator 115 determines whether or not the extracted value patterns include a value pattern that does not exist in the actual RDF data stored in the RDF store 110 (step S102). If there is no value pattern that does not actually exist (step S102: No), the first generator 115 proceeds to step S104.

If there is a value pattern that does not actually exist (step S102: YES), the first generation unit 115 extracts the value pattern that actually exists by removing the value pattern that does not actually exist from the extracted value patterns. (step S103).

Next, the first generating unit 115 assigns identifiers to actually existing value patterns, and generates an identifier correspondence table representing pattern identifiers corresponding to each value pattern (step S104).

Next, the first generating unit 115 divides the generated identifier correspondence table into subject-based search, predicate-based search, and object-based search to create divided identifier correspondence tables 231 to 233 . Next, the first generator 115 stores the split identifier correspondence tables 231 to 233 in the RDF store 110 (step S105). Further, first generation unit 115 notifies second generation unit 116 of completion of generation of division identifier correspondence tables 231 to 233 .

The second generation unit 116 that has received the notification of completion of generation of the divided identifier pair tables 231 to 233 acquires from the RDF store 110 all the RDF data and the identifier correspondence table contained in the RDF store 110 . Next, the second generation unit 116 acquires a value pattern that combines the subject and predicate of each RDF data, a value pattern that combines the predicate and object, and a value pattern that combines the subject and object. . Then, the second generating unit 116 acquires pattern identifiers corresponding to the acquired value patterns from the divided identifier correspondence tables 231 to 233. FIG. Next, the second generating unit 116 generates an RDF data table with identifier by adding the acquired pattern identifier to each piece of RDF data in the correspondence table representing triple correspondence (step S106).

The NameNode 112 acquires the RDF data table with identifier from the RDF store 110 . Next, the namenode 112 determines the datanode 121 in which each block containing the data registered in the identifier-attached RDF data table is placed. Then, the namenode 112 transmits each block including a part of row data of the identifier-attached RDF data table to each of the datanodes 121 determined as allocation destinations, and executes data distribution allocation (step S107).

Next, a flow of MapReduce processing according to the third embodiment will be described with reference to FIG. 19 . FIG. 19 is a flowchart of MapReduce processing according to the third embodiment.

The SPARQL processing unit 118 receives an input of a SPARQL query execution command from the job client 30 . The SPARQL processing unit 118 then executes the SPARQL query (step S201).

Next, the SPARQL processing unit 118 converts the SPARQL query into a job for MapReduce processing (step S202).

Next, the SPARQL processing unit 118 acquires the identifier correspondence table from the HDFS 111. FIG. Then, the SPARQL processing unit 118 parses the input query and determines whether or not there is a value pattern corresponding to the value patterns registered in the identifier correspondence table (step S203). If there is no corresponding value pattern (step S203: No), the SPARQL processing unit 118 returns to the job client 30 a search result indicating that the matching result for such a value pattern is 0, and ends the SPARQL query execution processing. do.

On the other hand, if there is a corresponding value pattern (step S203: Yes), the SPARQL processing unit 118 executes MapReduce processing. The Spark processing unit 131 receives an instruction from the SPARQL processing unit 118 and instructs the job tracker 114 to execute MapReduce processing. The job tracker 114 confirms the metadata DB 113 and selects a slave server 12 to perform the MapReduce process. The job tracker 114 then divides the MapReduce process into block-based Map tasks and transmits them to the selected slave server 12 . The task tracker 123 receives Map tasks from the job tracker 114 . The task tracker 123 then instructs the MapReduce processing unit 124 to execute the acquired Map task. The MapReduce processing unit 124 receives an instruction to execute a Map task from the task tracker 123 . Then, the memory management unit 243 acquires a table corresponding to the search criteria to be executed by the Map task from among the division identifier correspondence tables 231 to 233 (step S204). Here, a case where the division identifier correspondence table 231 is selected will be described.

Next, the memory management unit 243 converts the selected divided identifier correspondence table 231 and the identifier-attached RDF data table stored in the HDFS 122b into an RDD and develops it on the memory 126 (step S205).

The Map task execution unit 241 uses the divided identifier correspondence table 231 and the identifier-attached RDF data table developed on the memory 126 to execute the Map process specified by the Map task (step S206).

Next, the Map task execution unit 241 shuffles the processing results of the Map processing so that they are collected for each key and distributes them to the Map task execution unit 241 of each slave server 12 . Furthermore, the Map task execution unit 241 sorts the data distributed to the own device by shuffling so that the data are collected for each key (step S207). The Map task execution unit 241 then stores the sorted data in the memory 126 .

The Reduce task execution unit 242 executes a designated Reduce process on the data stored in the memory 126 by the Map task execution unit 241 (step S208).

After that, the Reduce task execution unit 242 stores the result of the MapReduce processing in the memory 126 . The memory management unit 243 acquires the execution result of the MapReduce process stored in the memory 126 , converts it into a data format for storage in the HDFS 111 , and stores it in the HDFS 122 . The task tracker 123 transmits the execution result of the MapReduce process stored in the HDFS 122 to the job tracker 114 of the master server 11 . The job tracker 114 collects the execution results of the MapReduce processing sent from each slave server 12 . The job tracker 114 then combines the execution results of the MapReduce process. The job tracker 114 then transmits the execution result of the combined MapReduce processing to the SPARQL processing unit 118 via the MapReduce processing unit 119 . The SPARQL processing unit 118 receives the execution result of the combined MapReduce process, and converts the received data into RDF format (step S209). After that, the SPARQL processing unit 118 transmits the result of the MapReduce process converted into the RDF format to the job client 30 as the execution result of the SPARQL query.

Here, in the present embodiment, the case of using Spark as distributed in-memory processing has been described, but other distributed in-memory processing may be used. The information processing system 1 may be configured to selectively execute MapReduce processing using distributed in-memory processing and MapReduce processing not using distributed in-memory processing described in the first embodiment. Furthermore, in the present embodiment, the configuration using Spark for the MapReduce processing described in the first embodiment has been described, but the configuration of the second embodiment can also be applied.

As described above, the Hadoop cluster according to this embodiment uses one of the identifier correspondence tables created according to the search target when executing the MapReduce process using Spark. As a result, a reduction in processing speed can be avoided by reducing the amount of data read into the memory and ensuring a sufficient memory capacity to be allocated to processing. In addition, since the number of entries included in the identifier correspondence table is reduced, it is possible to improve the efficiency of searching graph data. Furthermore, the efficiency of the MapReduce processing can be improved by executing the MapReduce processing by distributed in-memory processing.

The master server 11 and slave server 12 according to each of the embodiments described above can be realized by a computer having a hardware configuration as shown in FIG. 13, for example. The function of the Spark processing unit 131 is realized by the CPU 91 and the memory 92 when the master server 11 is realized by the computer 90 . Further, when the slave server 12 is implemented by the computer 90 , the MapReduce processing unit 124 is implemented by the CPU 91 and the memory 92 .

1 Information Processing System 10 Hadoop Cluster 11 Master Server 12 Slave Server 20 HDFS Client 30 Job Client 111 HDFS
112 name node 113 metadata DB
114 Job Tracker 115 First Generator 116 Second Generator 117 RDF Controller 118 SPARQL Processor 119 MapReduce Processor 121 Data Node 122 HDFS
123 Task Tracker 124 MapReduce Processing Unit 125 SSD
126 memory 131 Spark processing unit 241 Map task execution unit 242 Reduce task execution unit 243 Memory management unit

Claims (9)

Extracting two elements from data having three elements, generating a first table in which an identifier having a data size smaller than the extracted two elements is associated with the extracted two elements,
generating a second table in which the identifier is added to the three-element table;
dividing and arranging the second table in a plurality of processing devices;
When retrieving, the first table is used to retrieve the identifier, and the retrieved identifier is used in each of the processing units for a portion of the second table located in each of the processing units. to search,
A search processing program for causing a computer to execute a process of outputting rows extracted from the second table by the search. 2. The search processing program according to claim 1, wherein said second table is a table obtained by adding said identifier to said correspondence table of said three elements. 2. The search processing program according to claim 1, wherein a correspondence table is generated in which each element out of the three elements is associated with the identifier. The data having the three elements is graph data,
4. The search processing program according to any one of claims 1 to 3, wherein the search is a search using data in key=value format. 5. The method of claim 1, further causing the computer to perform a process of aggregating the output lines based on any one of the three elements and arranging them in different processing units according to the criteria. A search processing program according to any one of the above. 6. The search processing program according to claim 5, wherein each of said processing devices executes a predetermined process on the aggregated rows based on said reference elements. dividing the first table for each combination of the two elements;
Selecting one of the divided first tables according to a search target, and executing the search by distributed in-memory processing using the selected divided first table. 7. The search processing program according to any one of claims 1 to 6, characterized by: Extracting two elements from data having three elements, generating a first table in which an identifier having a data size smaller than the extracted two elements is associated with the extracted two elements,
generating a second table in which the identifier is added to the three-element table;
dividing and arranging the second table in a plurality of processing devices;
When retrieving, the first table is used to retrieve the identifier, and the retrieved identifier is used in each of the processing units for a portion of the second table located in each of the processing units. to search,
A search processing method, characterized by outputting a row extracted from the second table by the search. a first generating unit that extracts two elements from data having three elements and generates a first table in which an identifier having a data size smaller than the extracted two elements is associated with the extracted two elements;
a second generation unit that generates a second table in which the identifier is added to the three-element table;
an arrangement unit that divides and arranges the second table in a plurality of processing devices;
When retrieving, the first table is used to retrieve the identifier, and the retrieved identifier is used in each of the processing units for a portion of the second table located in each of the processing units. a search unit that searches for
and an output unit that outputs a row extracted from the second table by the search performed by the search unit.

Images