WO2013179451A1

WO2013179451A1 - Parallel data processing system, computer, and parallel data processing method

Info

Publication number: WO2013179451A1
Application number: PCT/JP2012/064149
Authority: WO
Inventors: 和生合田; 山田　浩之; 喜連川　優; 信男河村; 藤原　真二; 茂木　和彦
Original assignee: 国立大学法人東京大学; 株式会社日立製作所
Priority date: 2012-05-31
Filing date: 2012-05-31
Publication date: 2013-12-05
Also published as: US20150113535A1; EP2857975A1; EP2857975A4; US9841989B2; JPWO2013179451A1; JP5881025B2

Abstract

A parallel data processing system having a parallel data processing execution unit for reading data from a data group, which includes a first data group including a plurality of first data and a second data group including a plurality of second data, and executing a process. The parallel data processing execution unit: (A) reads a first data from the first data group, and acquires a first value from the first data on the basis of first format information acquired from an application; (B) generates, on the basis of first reference information acquired from the application, one or more threads for reading from the second data group each of one or more second data corresponding to the first value; (C) executes (A) and (B) with respect to one or more first data of the first data group; and (D) executes a plurality of threads in parallel.

Description

Parallel data processing system, computer, and parallel data processing method

The present invention relates to parallel data processing technology.

In the activities of companies, etc., it is becoming extremely important to use a large amount of data related to operations strategically. As an information system for using a large amount of data, a parallel data processing system such as Hadoop is used. The parallel data processing system is disclosed in Patent Document 1, for example.

US Pat. No. 7,756,919

In parallel data processing systems such as Hadoop, the entire data set stored in storage or the like (for example, a file stored in a file system) is basically read and processed. For example, even when an application (job) has selectivity for data to be processed (that is, when some records of a data set stored in a storage or the like are to be processed), the data The entire set needs to be read, and data processing is not necessarily efficient, and thus the time required for data processing may be increased. Further, as the data set becomes larger, the time required to read the entire data set becomes longer, and thus the time required for data processing may become longer.

Therefore, an object of the present invention is to shorten the time for data processing.

A parallel data processing system included in one computing node in a computer system that executes data processing in parallel in a plurality of computing nodes (for example, a computer) includes a first data group including a plurality of first data and a plurality of second data. A parallel data processing execution unit that reads data from the data group including the second data group and executes the process. The parallel data processing system may be, for example, a system module in the first and second embodiments described later. The first data in the first data group may correspond to the second data in the second data group. For example, the first data group may be an index of the second data group. In this case, the first data may include an index key value of the second data and a reference to one or more second data corresponding to the index key value.

The parallel data processing execution unit
(A) Reading the first data from the first data group, acquiring the first value from the first data based on the first format information acquired from the application,
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data in the first data group;
(D) A plurality of the threads are executed in parallel.

The parallel data processing system may further include a reception unit that receives processing instructions from the application. The instruction from the application generally defines the procedure, but the parallel data processing execution unit receives the instruction from the application and executes (A) to (D), so that the instruction from the application is Even if the procedure is defined, the parallel data processing system can execute out-of-order processing independent of the procedure.

According to the present invention, the calculation node can read data related to data processing in parallel. Thereby, it is expected that the throughput of data reading is improved, and therefore, the time for data processing is shortened.

FIG. 1A illustrates a job execution model according to the first embodiment. FIG. 1B illustrates a job execution model according to the second embodiment. FIG. 2A illustrates a configuration of a calculation node according to the first embodiment. FIG. 2B illustrates a configuration of a calculation node according to the second embodiment. FIG. 3A illustrates a first configuration example of the computer system according to the embodiment. FIG. 3B illustrates a second configuration example of the computer system according to the embodiment. FIG. 4A shows a flow of map task execution processing according to the first embodiment. FIG. 4B shows a flow of task execution processing according to the second embodiment. FIG. 5A is a diagram for explaining the input data and the process for the input data according to the first embodiment. FIG. 5B shows an example of a format and reference according to the first embodiment. FIG. 5C is an example of a schematic diagram illustrating thread generation and thread execution. FIG. 5D is a first diagram for explaining the input data and the process for the input data according to the second embodiment. FIG. 5E is a second diagram for explaining the input data and the process for the input data according to the second embodiment. FIG. 5F shows an example of a format according to the second embodiment. FIG. 5G illustrates an example of a reference method according to the second embodiment. FIG. 5H illustrates an example of a catalog according to the second embodiment. FIG. 6A is an example of a schematic diagram for explaining a session during data acquisition. FIG. 6B is an example of a schematic diagram for explaining a blocked session at the time of data acquisition according to the modification. FIG. 7A shows a flow of a record acquisition process in a calculation node that acquires a record according to the first embodiment. FIG. 7B shows a flow of record acquisition processing in a calculation node that manages records according to the first embodiment. FIG. 7C illustrates the configuration of the data read request management table and the remote record acquisition request management table according to the first embodiment. FIG. 7D illustrates a flow of record acquisition processing in a calculation node that acquires records according to the modification of the first embodiment. FIG. 7E shows the flow of a record acquisition process in a calculation node that manages records according to a modification of the first embodiment. FIG. 7F illustrates a configuration of a blocked remote record acquisition request management table according to a modification of the first embodiment. FIG. 8A shows a configuration of a node-level resource constraint management table according to the first embodiment. FIG. 8B illustrates a configuration of a job level resource constraint management table according to the first embodiment. FIG. 8C illustrates a configuration of a process level resource constraint management table in the overall supervisor process according to the first embodiment. FIG. 8D illustrates a configuration of a process level resource constraint management table in the supervisor process of each node according to the first embodiment. FIG. 8E illustrates a configuration of a task level resource constraint management table in the overall supervisor process according to the first embodiment. FIG. 8F illustrates a configuration of a task level resource constraint management table in the supervisor process of each node according to the first embodiment. FIG. 8G shows a flow of the overall resource constraint management process according to the first embodiment. FIG. 8H illustrates a flow of resource constraint management processing in each node according to the first embodiment. FIG. 9A illustrates a first example of tasks according to the first embodiment. FIG. 9B illustrates an example of thread generation in the task illustrated in the first example according to the first embodiment. FIG. 9C illustrates an example of thread generation in the task illustrated in the first example according to the first embodiment. FIG. 9D illustrates a second example of the task according to the first embodiment. FIG. 9E illustrates an example of thread generation in the task illustrated in the second example according to the first embodiment.

Hereinafter, some embodiments and modifications will be described with reference to the drawings. The present invention is not limited to the following description.

First, the job execution model according to the first embodiment will be described.

FIG. 1A shows a job execution model according to the first embodiment. In FIG. 1A, one solid circle represents one process (for example, a map process or a reduce process). One dashed circle in the process indicates one task. One rounded square in the task indicates one thread.

According to the job execution model, a job is executed by a plurality of computation nodes connected via a network. For example, a supervisor process in a compute node that supervises the entirety of multiple compute nodes (hereinafter referred to as a supervisor supervisor process) distributes application code to all compute nodes that participate in job execution, and the supervisor supervisor process performs each computation. Allocate processes such as map process and reduce process to the supervisor process of the node. The supervisor process of each compute node generates a process based on the instruction of the supervisory supervisor process. Further, the generated process generates a task based on an instruction from the supervisory supervisor process. Each computation node executes the job by executing the map operation included in the application by executing the processes and tasks generated in this way. The supervisory supervisor process may be any one of a plurality of supervisor processes in a plurality of computing nodes (that is, any one supervisor process may also serve as the supervisory supervisor process) or a plurality of supervisor processes. A dedicated process functioning as a dedicated supervisory supervisor process prepared separately from the process may be used. In addition, the process allocation is not limited to the above, and the supervisor supervisor process may instruct the supervisor process of each computation node to perform the process. The process and task generation is not limited to the above, and the supervisor supervisor process directly performs the process. Also good.

The parallel data processing system executes the map process according to the job execution model, and stores the input data set # 1 (that is, the first data set) and the input data set # 2 (that is, the second data set) stored in the storage. ) And a map operation are executed, and the result record is written to an intermediate data set stored in the storage. Further, the reduction process is executed, the result record written in the intermediate data set is input, the reduction operation is executed, and the result record is written in the output data set # 1. Each of the input data set, the intermediate data set, and the output data set is a collection of a plurality of records, and may be structured by some data structure or may not be structured. For example, the input data set # 2 may be a set of one or more files.

The input data set # 1 only needs to correspond to the record of the input data set # 2. For example, the input data set # 1 may be an index of the input data set # 2. Each record of the input data set # 1 includes a predetermined index key value of the record of the input data set # 2 and a reference indicating one or more records of the input data set # 2 corresponding to the index key value. Good. In this case, the reference may include a storage position where the record of the input data set # 2 can be specified on the storage device, or the record can be specified on the data structure provided for storing the input data set # 2. A unique index key may be included. The input data set # 1 may have records corresponding to all the records of the input data set # 2, or only records corresponding to some of the records of the input data set # 2. Also good. Further, the input data set # 1 may not be an index of the input data set # 2. For example, the input data set # 1 and the input data set # 2 are connectable, the input data set # 1 is simply a collection of records, and the input data set # 2 is a collection of records, depending on the data structure. A record of the input data set # 2 that is structured by a certain index key and has an index key value corresponding to the value of a certain item in a certain record of the input data set # 1 may be read. . Further, the input data set # 1 and the input data set # 2 may belong to the same data structure. For example, the input data set # 1 may be a set of internal nodes that constitute a B-tree, and the input data set # 2 may be a set of leaf nodes that constitute the same B-tree. Further, the input data set # 1 is a set of nodes at a certain level constituting the B-tree, and the input data set # 2 is a node at the next level (referenced from the input data set # 1) constituting the same B-tree. Or a set of Further, the input data set may be composed of three or more. For example, there may be another input data set corresponding to the record of the input data set # 2.

In the map process, data processing is executed by the processor of each calculation node according to map operation, division operation, format # 1, format # 2, reference method, conditions, and the like. Map operation, division operation, format # 1, format # 2, reference method, condition, and the like are program codes included in the application stored in the calculation node. The map operation is a program code that defines a process to be applied to the read record. For example, the map operation generates a result record including a key and value pair from the read record. A division operation is a program code that is executed when a map operation is executed and then a reduction process for delivering the execution result is determined. For example, a division operation is a hash operation on a key included in a result record of a map operation. May include. Format # 1 is a program code that defines a format for interpreting records of input data set # 1. Format # 2 is a program code that defines a format for interpreting the record of input data set # 2. The reference method is a program code that defines a method for acquiring the record of the input data set # 2 in accordance with the reference of the record of the input data set # 1. The condition is a program code that defines the requirements for the record to be subjected to the map operation among the records stored in the input data set # 1 and / or the input data set # 2. The condition may be executed when all or part of the record is read from either the input data set # 1 or the input data set # 2, or all or part of the condition is mapped to the read record. It may be executed when inputting to the calculation. The program code may be an instruction that can be executed by a processor generated by compilation or the like, an instruction that can be converted into an instruction that can be executed by the processor by an execution processing system, or the like. It may be a declaration that can generate an instruction that can be executed by the processor by an execution processing system or the like. Furthermore, it may be a combination of these and may further include other information. The instruction and the declaration may be a byte string that can be interpreted by a processor, a compiler, an execution processing system, or the like, or may be described in a source code or the like.

Specifically, the processor of the computation node executes a task in the map process, reads the record from the input data set # 1 in the task, interprets the record using the format # 1, and based on the condition, It is determined whether the record satisfies the requirement, and a reference to the input data set # 2 is acquired from the record that satisfies the requirement. At this time, the input data set # 1 may be divided into a plurality of chunks in advance. For example, the input data set # 1 may be composed of a plurality of files, and the supervisor process assigns a different task to each file, and each task may read a record from the assigned file. . Alternatively, the input data set # 1 may be separable as if it is composed of a plurality of chunks at the time of execution. For example, the input data set # 1 is composed of a single file and includes information indicating a plurality of non-overlapping areas in the file, and the supervisor supervisor process regards each area as the aforementioned chunk. Then, another task may be allocated, and each task may read a record from the allocated area. The information indicating the area may be given in advance, or may be determined at the time of execution by a supervisory supervisor process. In addition, even if there is an overlap between an area and another area, there may be provided means for preventing a task assigned to one area from reading an overlapped portion when reading. Further, when reading records from the input data set # 1, the input data set # 1 is configured with a data structure that allows selective access to records that satisfy the requirements specified in the conditions. If they are arranged (for example, arranged or constituted by B-trees, etc.), selective access may be performed using the requirements defined in the conditions.

Next, the processor creates a thread to read the record from the input data set # 2 based on the reference included in the record of the input data set # 1. When the record of the input data set # 1 includes a plurality of references, the processor generates a plurality of threads. For example, a thread may be generated for each reference included in the record of the input data set # 1. These threads are executed in parallel by the processor. Further, the processor reads the record of the input data set # 2 by reference based on the reference method in the thread to be executed. The processor interprets the read record using the format # 2, and determines whether the record satisfies the requirement based on the condition. The processor performs a map operation on the record that satisfies the requirement, and further determines a reduce process to which the execution result of the map operation is to be sent by executing a division operation. The processor outputs so that the determined reduce process can receive the execution result of the map operation. Specifically, the execution result is stored in the intermediate data set.

In the reduce process, data processing is executed by the processor of each computation node in accordance with the reduce operation or the like. The reduce operation is a program code included in the application stored in the calculation node. A reduction operation is a program code that defines the processing applied to records in an intermediate data set. For example, a result is obtained by aggregating result records (consisting of key / value pairs) generated by a map operation according to a key. A record is generated.

Specifically, the processor acquires the execution result record of the map operation from the intermediate data set, performs the reduce operation using the record as an input, and stores the execution result record in the output data set # 1.

The map operation, the division operation, the format # 1, the format # 2, the reference method, the condition, and the reduce operation need not all be specified by the application, and only a part thereof may be specified. For those that are not specified, predetermined handling may be performed. For example, when the reduction operation is not defined (that is, when the reduce process is not defined), the output of the map process may be regarded as the output of the job.

In the map process and / or the reduce process, data processing may be executed by the processor of each calculation node further according to the comparison operation, and in the map process, further according to the aggregation operation. The comparison operation and the aggregation operation are program codes included in the application stored in the calculation node. The comparison operation temporarily aligns the result record / input record when writing the result record of the split operation to the intermediate data set in the map process or when reading the input record of the reduce operation from the intermediate data set in the reduce process. Therefore, this is for defining the order between the records. For example, the comparison operation compares the magnitudes of the key values of the result records (consisting of key-value pairs) generated by the map operation. The comparison operation may not be specified by the application, may be specified for either the map process or the reduce process, the same comparison operation may be specified for both processes, or both Different comparison operations may be defined for these processes. The aggregation operation is for once aggregating the result records when writing the result records of the division operation to the intermediate data set in the map process. For example, the aggregation operation is to generate a result record by aggregating according to the key of the result record (consisting of key and value pairs) generated by the map operation. Aggregation operations may not be specified by the application. A comparison operation and an aggregation operation may be defined for the map process. In this case, for example, an aggregation operation may be performed on the result records arranged according to the comparison operation.

In the map process, records from multiple different data sets may be entered in parallel or sequentially. For example, in the above example, the record is read from the input data set # 2 referenced from the input data set # 1 and used as the input to the map operation. However, the input referenced from the input data set # 1-2 is additionally provided. A record may be read from the data set # 2-2 and used as an input to the map calculation. At this time, the same format may be used as the format # 1, the format # 2, the reference method, and the conditions, or different formats may be used.

Figure 1A shows a job execution model consisting of the map process and the reduction process shown in the document Jeffrey Dean, Sanjay Ghemawat: MapReduce: Simplified Data Processing on Large Clusters. Proceedings of OSDI 2004, pp. 137-15 (2004). However, other job execution models may be used. For example, FIG. 1A shows an example in which the input data set has two stages, but the input data set may have two or more stages. Further, FIG. 1A shows a job execution model composed of two stages of processes, that is, a map process and a reduce process. However, a job execution model composed of three or more stages of processes may be used. In FIG. 1A, only the first map process reads records from the input data set stored in the storage. For example, other stage processes read records from the storage input data set. May be. Also, in FIG. 1A, the map process once writes the result record to the intermediate data set, and the reduce process reads the result record from the intermediate data set, thereby passing the record from the map process to the reduce process. However, when the map process is executed, the map operation result record is transmitted through the network without being stored in the intermediate data set, and when the reduce process is executed, the map operation result record is received through the network. Also good. The map process and the reduce process may be executed in parallel. In addition, when executing the reduce process, the result record of the reduce operation may be output to the console or the printer without being written to the output data set, or may be transmitted to another computer or the like via the network. . The map process may send the result record over the network to the reduce process, and the reduce process may receive the result record over the network. Examples of job execution models extended in this way are the documents Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly: Dryad: distributed data-parallel programs from sequential building blocks. Proceedings of EuroSys 59, 72 (2007) and references Vinayak R. Borkar, Michael J. Carey, Raman Grover, Nicola Onose, Rares Vernica: Hyracks: A flexible and extensible foundation for data-intensive computing. Proceedings 162 ) Is disclosed in.

Next, a job execution model according to the second embodiment will be described.

FIG. 1B shows a job execution model according to the second embodiment. In addition, FIG. 1B is described similarly to FIG. 1A. The job execution model according to the second embodiment is an extension of the job execution model according to the first embodiment, and the part described with respect to the first embodiment is applied to the second embodiment without being described again.

According to this job execution model, a job is executed by a plurality of computation nodes connected via a network. The job illustrated in FIG. 1B reads records from the input data set # 1, the input data set # 2, the input data set # 3, and the input data set # 4 by executing the stage # 1 process, and inputs the records. As a result, the stage calculation # 1 is executed, and the execution result record is transmitted to the stage # 2 process. Further, by executing the stage # 2 process, the execution result record is received, and the stage calculation # 2 is executed by using this as an input, and the execution result record is transmitted to the stage # 3 process. Further, the execution result record is received by executing the stage # 3 process, and the stage calculation # 3 is executed by using this as an input, and the execution result record is transmitted to the stage # 4. In addition, by executing the stage # 4 process, the execution result record is received, the records are read from the input data set # 5 and the input data set # 6, and the stage # 4 operation is executed using these as input records. The execution result record is stored in the output data set # 1. As in the first embodiment, between processes at different stages, data may be transferred via an intermediate data set, or data may be transmitted / received via a network. Also, different stages of processes may be executed in parallel.

The input data set # 1 may be an index of the input data set # 2. For example, each record of the input data set # 1 has a reference value indicating a predetermined index key value of the record of the input data set # 2 and one or more records of the input data set # 2 corresponding to the index key value. May include. Similarly, the input data set # 3 may be an index of the input data set # 4. Each record of the input data set # 3 includes a predetermined index key value of the record of the input data set # 4 and a reference indicating one or more records of the input data set # 4 corresponding to the index key value. Good. Further, the predetermined item of the record of the input data set # 2 may include a reference indicating one or more records of the input data set # 3. That is, it may be possible to associate one or more records of the input data set # 3 with a certain record of the input data set # 2. As in the first embodiment, the reference may include a storage position where the record of the reference input data set can be specified on the storage device, or the record can be specified in the data structure constituting the reference input data set. A unique index key may be included. Further, the reference source input data set may have records corresponding to all the records of the reference destination input data set, or only records corresponding to some records of the reference destination input data set. You may do it.

The input data set # 5 may be an index of the input data set # 6. Each record of the input data set # 5 includes a predetermined index key value of the record of the input data set # 6 and a reference indicating one or more records of the input data set # 6 corresponding to the index key value. Good.

In the stage # 1 process, stage operation # 1, division operation # 1, format # 1, format # 2, format # 3, format # 4, reference method # 1, reference method # 2, reference method # 3, condition # 1. In accordance with the build method # 1, etc., processing is executed by the processor of each computation node. Stage operation # 1, division operation # 1, format # 1, format # 2, format # 3, format # 4, reference method # 1, reference method # 2, reference method # 3, condition # 1, build method # 1, etc. Is the program code included in the application stored in the compute node. Stage operation # 1 is a program code that defines the process applied to the read record. For example, it may be a map operation, a reduce operation, or another operation. Also good. The division operation # 1 is a program code that is executed when determining the process of the subsequent stage that delivers the operation result. For example, the division operation may include a hash function for the input. Format # 1, Format # 2, Format # 3 and Format # 4 are formats for interpreting the records of input data set # 1, input data set # 2, input data set # 3 and input data set # 4, respectively. This is a program code to be defined. Reference method # 1, reference method # 2 and reference method # 3 are respectively referred to the records of input data set # 1, input data set # 2 and input data set # 3, and input data set # 2 and input data set #. 3 and a program code that defines a method for acquiring records of the input data set # 4. Condition # 1 is a procedure that defines the condition of the record that should be the target of stage operation # 1. The condition # 1 may be executed when all or a part of the condition # 1 reads a record from any one of the input data set # 1, the input data set # 2, the input data set # 3, and the input data set # 4. , All or a part thereof may be executed when the read record is input to the map operation. Build method # 1 is a program that defines a method for generating records to be input to stage operation # 1 from records read from input data set # 1, input data set # 2, input data set # 3, and input data set # 4 Code.

More specifically, the processor of the computation node executes a task in the stage # 1 process, reads a record from the input data set # 1 in the task, interprets the record using the format # 1, and satisfies the condition # 1. Based on the above, it is determined whether or not the record satisfies the requirement, and a reference to the input data set # 2 is acquired from the record that satisfies the requirement.

Next, the processor creates a thread to read the record from the input data set # 2 based on the reference included in the record of the input data set # 1. When the record of the input data set # 1 includes a plurality of references, the processor generates a plurality of threads. For example, a thread may be generated for each reference included in the record of the input data set # 1. These threads are executed in parallel by the processor. Furthermore, the processor reads a record from the input data set # 2 by reference based on the reference method # 1 in the thread to be executed. The processor interprets the record read using the format # 2, and determines whether the record satisfies the requirement based on the condition # 1. Further, the processor obtains a reference to the input data set # 3 from the record that satisfies the requirement. Here, the processor generates a thread that performs processing by referring to the input data set # 3 using the reference. If there are multiple references, the processor creates multiple threads. These threads are executed in parallel by the processor.

Next, the processor creates a thread to read the record from the input data set # 3 based on the reference contained in the record of the input data set # 2. When the record of the input data set # 2 includes a plurality of references, the processor generates a plurality of threads. For example, a thread may be generated for each reference included in the record of the input data set # 2. These threads are executed in parallel by the processor. Further, the processor reads a record from the input data set # 3 by reference based on the reference method # 2 in the thread to be executed. The processor interprets the read record using the format # 3, and determines whether the record satisfies the requirement based on the condition # 1. In addition, the processor obtains a reference to the input data set # 4 from the record that satisfies the requirement.

Next, the processor creates a thread to read the record from the input data set # 4 based on the reference contained in the record of the input data set # 3. When the record of the input data set # 3 includes a plurality of references, the processor generates a plurality of threads. For example, a thread may be generated for each reference included in the record of the input data set # 3. These threads are executed in parallel by the processor. Further, the processor reads a record from the input data set # 4 by reference based on the reference method # 3 in the executing thread. The processor interprets the read record using the format # 4, and determines whether the record satisfies the requirement based on the condition # 1. The processor generates a record to be input to the stage calculation # 1 based on the build method # 1 with respect to the record that satisfies the requirements, and executes the stage calculation # 1 using the record as an input. Further, the division operation # 1 is performed on the execution result record of one or more stage operations # 1, thereby determining the subsequent stage # 2 process to which the execution result record is to be sent. The processor outputs so that the determined subsequent stage process can receive the execution result record. Specifically, the execution result is stored in an intermediate data set or sent to a subsequent stage process on the network.

In the stage # 2 process, processing is executed by the processor of each computation node in accordance with stage operation # 2 and division operation # 2. Stage calculation # 2 and division calculation # 2 are program codes included in the application stored in the calculation node. For example, stage operation # 2 is a program code that defines processing applied to a record acquired from an intermediate data set or a record sent over a network, and may be a map operation or a reduce operation, for example. It may be other calculations. The division operation # 2 is a program code that is executed when determining the process of the subsequent stage # 3 that delivers the execution result. For example, the division operation # 2 may include a hash function for the input.

Specifically, the processor of the computation node executes a task in the stage # 2 process, acquires a record passed from the stage # 1 process in the task, executes the stage calculation # 2, and further divides By performing the calculation # 2, the process of the subsequent stage # 3 to which the execution result record of the stage calculation # 2 is to be sent is determined. The processor outputs so that the process of the determined subsequent stage can receive the execution result record of the stage operation # 2. Specifically, the execution result is stored in an intermediate data set or sent to a subsequent stage process on the network.

In the stage # 3 process, processing is executed by the processor of each computation node according to the stage operation # 3 and the division operation # 3. Stage calculation # 3 and division calculation # 3 are program codes included in the application stored in the calculation node. For example, the stage operation # 3 is a program code that defines processing applied to a record acquired from an intermediate data set or a record sent over a network, and may be a map operation or a reduce operation, for example. It may be other calculations. The division operation # 3 is a program code that is executed when determining the process of the subsequent stage # 4 that delivers the execution result. For example, the division operation # 3 may include a hash function for the input.

Specifically, the processor of the computation node executes a task in the stage # 3 process, acquires a record passed from the stage # 2 process in the task, executes the stage calculation # 3, and further divides By performing the calculation # 3, the process of the subsequent stage # 4 to which the execution result record of the stage calculation # 3 is to be sent is determined. The processor outputs so that the process of the determined subsequent stage can receive the execution result record of the stage operation # 3. Specifically, the execution result is stored in an intermediate data set or sent to a subsequent stage process on the network.

In the stage # 4 process, processing is executed by the processor of each calculation node in accordance with stage calculation # 4, format # 5, format # 6, reference method # 4, and the like. Stage operation # 3, division operation # 3, format # 5, format # 6, and reference method # 4 are program codes included in the application stored in the calculation node. For example, stage operation # 4 is a program code that defines processing applied to a record acquired from an intermediate data set or a record sent over a network, and may be a map operation or a reduce operation, for example. It may be other calculations. Format # 5 and format # 6 are program codes that define formats for interpreting the records of input data set # 5 and input data set # 6, respectively. The reference method # 4 is a program code that defines a method for acquiring the record of the input data set # 6 in accordance with the reference of the record of the input data set # 5.

More specifically, the computation node processor executes a task in the stage # 4 process, reads a record from the input data set # 5, interprets the record using the format # 5, and records the record. To obtain a reference to the input data set # 6.

Next, the processor creates a thread to read the record from the input data set # 6 based on the reference included in the record of the input data set # 5. When the record of the input data set # 5 includes a plurality of references, the processor generates a plurality of threads. For example, a thread may be generated for each reference included in the record of the input data set # 5. These threads are executed in parallel by the processor. Further, the processor reads a record from the input data set # 6 by reference based on the reference method # 4 in the thread to be executed. The processor interprets the read record using format # 6. Also, a record that is the execution result of stage calculation # 3 is acquired. Using these records as input, stage calculation # 4 is executed, and the execution result record is stored in output data set # 1.

Similar to the first embodiment, the stage operation # 1, the division operation # 1, the format # 1, the format # 2, the format # 3, the format # 4, the reference method # 1, the reference method # 2, the reference method # 3, and the condition # 1. Build method # 1, stage operation # 2, division operation # 2, stage operation # 3, division operation # 3, stage operation # 4, format # 5, format # 6, and reference method # 4 are all applications. It is not necessary to be specified, and only a part of it may be specified. For those that are not specified, predetermined handling may be performed.

As in the first embodiment, in each stage process, data processing may be executed by the processor of each calculation node in accordance with the comparison operation and the aggregation operation.

In the process of each stage, records from a plurality of different data sets may be input in parallel or sequentially.

The job execution model according to the first and second embodiments has been described above. The job execution model is not limited to them.

The common points regarding the job execution models according to the first and second embodiments are as follows, for example.

In parallel data processing systems such as Hadoop, data groups in the storage space are generally not structured. On the other hand, according to the first embodiment and the second embodiment, in the storage (for example, the file system), the first data set (for example, the input data set # 1) composed of a plurality of first records is Are associated with a second data set (for example, input data set # 2) composed of a plurality of second records. Specifically, for example, the first data set is an index for specifying a second record required for data processing in the second data set. That is, the first data set is structured, but the second data set may not be structured. Because the first data set is structured, the second data set associated with it is used for data processing in the second data set even if it is not structured. It becomes possible to specify and access the required second record. As another example, the first data set and the second data set have a relationship that allows respective records to be combined. That is, the first data set may not be structured, and the second data set may be structured. Since the second data set is structured, the second record corresponding to the record of the first data set associated with the second data set can be selectively extracted.

At this time, according to the first and second embodiments, information on the format of the first data set (for example, format # 1), information on the format of the second data set (for example, format # 2), The application prepares information including information related to a method for referring to the second record based on the record (for example, a reference method). The information is, for example, a program code. The information may be included in another program (for example, a system module described later) independently of each application. However, there are also benefits to the application being able to provide this individually. For example, a database management system (DBMS) manages a data group (table) in a storage space based on a relational model. Catalog information such as format and reference method is managed independently of the application. In this case, for example, the column type of the record is selected from types already prepared by the DBMS (for example, integers, character strings, etc.), but depending on the application, a type other than the type already provided by the DBMS may be selected. Thus, it may be desirable to interpret the column. The fact that an application can individually prepare information such as a format and a reference method may be recognized from the viewpoint of flexibility as a data processing system. In DBMS, it is generally required that all records in a data set (table) are strictly in accordance with given catalog information. However, even when a column is added to a record recorded after a specific date and time in a certain data set, in principle, it is necessary to add a column to all records in the data set (table). That is, it is necessary to add a column (for example, empty) even for a record before a specific date that does not require the column. Usually, such an operation requires a process of reconfiguring or reorganizing a data set (table). When the data set (table) becomes large, the time required for the process cannot be ignored. On the other hand, if the application can prepare information such as format and reference method, the application may prepare information such as format and reference method that is different for records before a specific date and records after a specific date. it can. That is, by changing the application without changing the data set itself, the data processing can be flexibly made compatible with the change of the record format, and an advantage may be recognized.

Further, according to the first and second embodiments, a program (for example, a system module) different from the application reads the first record from the first data set based on the job information of the data processing specified by the application. , Interpreting the record based on the job information, generating and executing a thread based on the job information, further reading the second record from the second data set in each thread, and defining the job information for the read record It is possible to execute the operation. The calculation node can execute data reading related to data processing in parallel according to job information defined by the application. Thereby, it is expected that the throughput of data reading is improved, and therefore, the time for data processing is shortened.

Hereinafter, Examples 1 and 2 will be described in detail. The second embodiment is an extension of the first embodiment, and the part described with respect to the first embodiment is applied to the second embodiment without being described again.

FIG. 2A shows a configuration of a calculation node according to the first embodiment. FIG. 2A shows a configuration of a computation node that performs data processing according to the job execution model shown in FIG. 1A.

The computing node 100 may be a computer such as a personal computer, a workstation, a server, or a mainframe. The computing node 100 may be an auxiliary computing device (for example, an image processing device (GPU) card) including an image processing device (GPU; Graphical Processing Unit) that is used by being attached to the computer. In addition, a virtual computer configured by virtual software, virtual hardware, or the like in the auxiliary arithmetic device may be used.

The computation node 100 has a communication interface, a storage device, and an arithmetic device connected to them. As the communication interface, for example, there are a NIC (Network Interface Card) 102 and an HBA (Host Bus Adapter) 103. Examples of the storage device include a storage 104 and a memory 105. The arithmetic device is, for example, a processor (CPU; Central Processing Unit) 101. In addition to the processor, the control device may include a hardware circuit that performs dedicated processing (for example, encryption or decryption). The processor 101, NIC 102, HBA 103, and memory 105 are connected via an internal bus 106. The storage 104 is connected to the HBA 103.

The processor 101 executes a computer program. The NIC 102 connects the network 200 and the calculation node 100. As a protocol for communication via the network 200, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) may be employed. The HBA 103 mediates input / output to the storage 104.

The storage 104 includes one or more nonvolatile storage media. The nonvolatile storage medium is, for example, a magnetic disk or a flash memory. The storage 104 may include a plurality of nonvolatile storage media, and may further include a RAID (Redundant ARRAY of Independent Disks) controller that configures a storage space from the nonvolatile storage media.

The memory 105 is, for example, a volatile storage medium (for example, DRAM (Dynamic Random Access Memory)), and stores a program executed by the CPU 101, data used by the program, and the like.

The memory 105 includes an application program (hereinafter referred to as an application) 110, a system module 120, a process manager 131, a task manager 132, a thread manager 133, a data reader / writer 140, a network manager 150, a storage manager 160, and an OS (Operating System) 170. Store. The system module 120, the process manager 131, the task manager 132, the thread manager 133, the data reader / writer 140, the network manager 150, and the storage manager 160 (hereinafter, individual program modules are collectively referred to as a module group) and the application 110 are static. Alternatively, it may be a library module that is dynamically linked and executed. In this case, an instruction from the application 110 or a mutual instruction between program modules in the module group depends on a call interface disclosed by the module group. Further, the module group may be a program that operates separately from the application 110. In this case, an instruction from the application 110 is based on means such as inter-process communication or shared memory.

The application 110 is a program that defines a job that reads an input data set stored in the storage 104, executes predetermined processing, and writes an output data set, and the calculation node executes the job by executing the application 110. To do. For example, the map 110a, the reduce operation 110b, the division operation 110c, the format 110e (format # 1), and the format 110f (format 110f) are used as the information that defines the job (hereinafter referred to as job information). # 2) and condition 110g.

The map calculation 110a is a program code that defines a process to be applied to the read record. For example, the map calculation 110a generates a result record including a key and value pair from the read record. The division operation 110c is a program code executed when the map operation is executed and then the reduction process for transferring the execution result is determined. For example, the division operation is a hash operation on a key included in the result record of the map operation. May be included. Format 110e (format # 1) is a program code that defines a format for interpreting records of input data set # 1. The format 110f (format # 2) is a program code that defines a format for interpreting the record of the input data set # 2. The reference method 110h is a program code that defines a method for acquiring the record of the input data set # 2 in accordance with the reference of the record of the input data set # 1. The condition 110g is a program code that defines the requirements for the record to be subjected to the map operation among the records stored in the input data set # 1 and / or the input data set # 2. Note that the application 110 does not have to define all the program codes described above. For those that are not specified, predetermined handling may be performed. For example, when the reduction operation is not provided, the reduction process is not executed, and the execution result record of the map operation in the map process is stored as it is in the output data set. Further, the application 110 may define a comparison operation (not shown) and an aggregation operation (not shown).

The system module 120 is a program module that operates independently of the application 110, but operates in cooperation with the application 110. In response to an instruction to execute a job from the application 110, the system module 120 performs a job according to job information defined in the application 110. Execute. The system module 120 includes an interface unit (not shown) that receives a job execution instruction from the application 110, an execution unit that includes a map function 121, a reduce function 122, and a supervisor function 123. The map function 121 is a program code (function) executed in the map process. The reduce function 122 is a program code (function) executed in the reduce process. The supervisor function 123 is program code that is executed in a supervisor process (including a supervisory supervisor process). The system module 120 accepts a job execution instruction from the application 110 in the interface unit, generates a process required to execute the job in the execution unit, executes each function in the process, and executes each function To generate and execute tasks and threads.

Details of the system module 120 will be described. The CPU 101 of each calculation node generates a supervisor process by the system module 120 and executes the supervisor function 123 in the supervisor process. The generation of the supervisor process described above may be performed automatically by, for example, starting a computation node in advance, or may be performed as necessary after starting execution of a job. One of the supervisor processes of the plurality of computation nodes 100 that execute the job also performs processing for supervising the plurality of computation nodes 100. A supervisor process that supervises a plurality of computation nodes 100 is referred to as a supervisory supervisor process. The supervisor supervisor process may be fixedly a supervisor process of one of the computation nodes 100, or may be a supervisor process of the computation node 100 selected from the plurality of computation nodes 100. Note that a computer other than the computation node 100 may be provided with a general supervisor process.

The supervisory supervisor process distributes the code of the application 110 to all the computation nodes 100 participating in the job execution, and assigns the process to each supervisor process. The supervisor process of each node generates a process based on the instruction of the supervisory supervisor process. The supervisory supervisor process performs processing to grasp the state of the supervisor process of all the computation nodes participating in job execution. Each supervisor process performs processing for grasping the execution state of a process related to job execution in its own computation node.

The process manager 131 manages memory resources required to execute a process based on an instruction from the system module. That is, process creation, deletion, execution status, and the like are managed. The task manager 132 manages memory resources required to execute a task based on an instruction from the system module. That is, task generation, deletion, execution status, and the like are managed. The thread manager 133 manages memory resources required to execute a thread based on an instruction from the system module. That is, it manages the generation, deletion, execution state, etc. of threads.

The data reader / writer 140 reads / writes data from / to the storage based on instructions from the system module 120. The data reader / writer 140 may be a file system, for example. For example, if the data reader / writer 140 needs to read / write data from / to the storage 104 of its own computing node 100 in order to perform the specified data read / write, the storage manager 160 causes the storage 104 to execute data read / write, When it is necessary to read / write data from / to the storage 104 of another computing node 100 connected via the network 200, the data is transferred to the storage 104 of another computing node 100 connected via the network 200 by the network manager 150. Read / write is executed. At this time, the data reader / writer 140 may temporarily cache data to be read / written using the memory resource of the memory 105.

The network manager 150 controls data communication with devices (for example, other computing nodes 100) connected via the network. The storage manager 160 controls input / output with the storage 104 of its own computing node 100. The OS 170 manages devices such as the NIC 102, the HBA 103, and the storage 104, and also manages the entire computing node 110.

In the present specification, embodiments and modifications of the invention may be described using a program or a part of its name (for example, an application, a system module, a process, a task, a thread) as a subject. In this case, the program and a part of the program are executed by the arithmetic device (for example, the processor 101) included in the calculation node 100, so that the determined processing is appropriately performed by the storage device (for example, the memory 105 and the storage 104). And / or using a communication interface (for example, NIC 102 or HBA 103), the subject in the description of the embodiments of the invention and the modifications may be interpreted as the arithmetic unit, the processor 101, or the calculation node 100. Good. In addition, the program or a part thereof may be executed by hardware, and in that case, the subject in describing the embodiment of the invention and the modified example is the hardware instead of or in addition to the processor 101. May be interpreted as Computer programs such as the system module 120, the process manager 131, the task manager 132, the thread manager 133, the data reader / writer 140, the network manager 150, the storage manager 160, etc. may be installed in the computing node 100 from the program source. The program source may be, for example, a storage medium that can be read by the computation node 100, or a computer that is connected to the computation node 100 so as to be communicable.

The computing node 100 may include a plurality of at least one element of the CPU 101, the NIC 102, and the HBA 103 from the viewpoint of performance and availability. The computing node 100 may include an input device (for example, a keyboard and a pointing device) (not shown) and a display device (for example, a liquid crystal display) (not shown). The input device and the display device may be integrated.

FIG. 2B shows a configuration of a calculation node according to the second embodiment. FIG. 2B shows a configuration of a computation node that performs data processing according to the job execution model shown in FIG. 1B.

The application 110 includes, as job information, for example, one or more stage operations 110j, one or more division operations 110k, one or more formats 110m, one or more conditions 110n, one or more reference methods 110o, and one or more. The build method 110p.

The stage operation 110j is a program code that defines a process applied to a record input in the process of each stage in the job execution model. For example, the stage operation 110j may be a map operation or a reduce operation. It may be another operation. In the case of the job execution model of FIG. 1B, four stage calculations of stage calculations # 1 to # 4 are included. The division operation 110k is a program code that is executed when determining the process of the subsequent stage that delivers the execution result record. For example, the division operation may include a hash operation on the input. The format 110m is a program code that defines a format for interpreting the records of each input data set. The application 110 that executes the job execution model shown in FIG. 1B stores a format 110m corresponding to each of the input data sets # 1 to # 6. The condition 110n is a program code that defines the requirements of the record that should be the target of the stage calculation. The reference method 110o is a program code that defines a method for acquiring a record from a reference input data set in accordance with the reference of a record of a certain input data set. The build method 110p is information indicating a method of creating an execution result record. Note that the application 110 does not have to define all the program codes described above. For those that are not specified, predetermined handling may be performed. Further, the application 110 may define a comparison operation (not shown) and an aggregation operation (not shown).

The system module 120 includes a generalized stage function 124 and a supervisor function 122. The generalized stage function 124 is program code (function) executed in each process shown in FIG. 1B.

FIG. 3A shows a first configuration example of the computer system according to the embodiment. FIG. 3B illustrates a second configuration example of the computer system according to the embodiment.

As shown in FIG. 3A, the computer system may include a plurality of calculation nodes 100 connected by a network 200 (for example, a local area network based on Ethernet (registered trademark)) via the NIC 102. Further, as shown in FIG. 3B, the computer system is connected to the network 200 via the NIC 102, and is further connected to one or more storages 400 via the network 300 (for example, a storage area network using FiberChannel) via the HBA 103. A plurality of calculation nodes 100 may be included. Further, the storage 400 may be connected to the network 200 instead of the network 300, and may perform input / output using a communication protocol (NFS or CIFS) for shared file access.

The storage 400 includes one or more nonvolatile storage media. The nonvolatile storage medium is, for example, a magnetic disk or a flash memory. The storage 104 may include a plurality of nonvolatile storage media, and may further include a RAID (Redundant ARRAY of Independent Disks) controller that configures a storage space from the nonvolatile storage media. All or part of the storage resources of the storage 400 may be handled in the same manner as the storage 104 included in the computing node 100.

FIG. 4A shows the flow of map task execution processing according to the first embodiment.

This map task execution process indicates a process executed in the map task executed in the map process in the job execution model shown in FIG. 1A.

The processor 101 of the computing node 100 executes the processing of steps S10 to S15 by executing one thread SL1 for reading the record of the input data set # 1 and executing the processing. This process is realized by the processor 101 executing mainly the map function 121.

In S10, the processor 101 acquires one record from the input data set # 1. Here, the input data set # 1 may be stored in the storage 104 of its own calculation node 100, or may be stored in the storage 104 of another calculation node 100. The record acquisition process for acquiring records from the input data set # 1 will be described later.

In S11, the processor 101 interprets the contents of each item included in the acquired record based on the format 110e (format # 1), and the input data set # 1 included in the condition 110g is obtained for the acquired record. By applying the condition for the record, it is determined whether or not the record satisfies the condition. If necessary, the process proceeds to S12. If not necessary, the process proceeds to S15 although not shown. In S11, a part of the condition 110g may be applied. If the condition 110g is not defined, the process may proceed to S12 as it is.

In S12, the processor 101 determines whether or not the acquired record has a reference to the input data set # 2. If the result of this determination is affirmative, S13 is performed, while if the result of this determination is negative, S15 is performed.

In S13, the processor 101 generates a thread SL2 for acquiring a record from the input data set # 2 and performing processing for one reference to the input data set # 2 of the acquired record.

In S14, the processor 101 determines whether or not there is a reference to the unprocessed input data set # 2 in the acquired record. If the result of this determination is affirmative, S13 is performed, while if the result of this determination is negative, S15 is performed. Thus, if there are a plurality of references to the input data set # 2 in the acquired record, the thread SL2 corresponding to the number of references is generated in S13. Note that when the resources necessary for generating the thread are insufficient, the generation of the thread SL2 may be temporarily suspended. At this time, one thread SL2 may be generated for each reference, or one thread SL2 may be generated for a plurality of (for example, a predetermined number) references.

In S15, the processor 101 determines whether there is another record in the input data set # 1. If the result of this determination is affirmative, S10 is performed. If the result of this determination is negative, this process is terminated and the thread SL1 that executed this process is stopped.

On the other hand, the thread SL2 generated by the thread SL1 in S13 is executed by the CPU 101. The processor 101 executes the processing of step S16 to step S19 by executing the thread SL2. In the present embodiment, the processor 101 can execute a plurality of threads (thread SL1, thread SL2, etc.) in parallel. The computing node 100 includes a plurality of processors 101, and a thread SL2 generated by one processor 101 may be executed by another processor 101. Note that the number of threads that can be executed in parallel is limited by the resources of the computing node 100 and the like.

In S16, the processor 101 acquires one record from the input data set # 2 using the reference acquired by the reference method 110h and the thread SL1.

In S17, the processor 101 interprets the contents of each item included in the acquired record based on the format 110f (format # 2), and the input data set # 2 included in the condition 110g for the acquired record. By applying the condition for the record, it is determined whether or not the record is necessary. If necessary, the process proceeds to S18. If not necessary, the process proceeds to S19 although not shown. In S17, a part of the condition 110g may be applied. If the condition 110g is not defined, the process may proceed to S18 as it is.

In S18, the processor 101 stores the acquired record in the calculation queue 180 of the main memory 105.

In S19, the processor 101 determines whether there is another record in the range indicated by the reference of the input data set # 2. If the result of this determination is affirmative, S16 is performed, while if the result of this determination is negative, this process is terminated and the thread SL2 that performed this process is terminated.

In S20, the processor 101 obtains one record from the computation queue 180, applies a map computation 110a to the record, executes a predetermined process on the record, and obtains the execution result. Output. At this time, the processor 101 may execute S20 in a thread different from SL2. There may be one or more threads that execute S20. The map calculation 110a may be applied by acquiring a plurality of records at once from the calculation queue 180. Instead of executing S18 in the thread SL2, the processor 101 may execute S20 after executing S17 in the thread SL2 and apply the map operation to the record of the result of S17. Here, as a method of outputting the execution result, for example, the execution result may be stored in the main memory 105, or the execution result may be passed to a reduce process that executes subsequent processing.

In FIG. 4A, a thread SL1 that reads records from the input data set # 1 and performs processing thereof and a thread SL2 that reads records from the input data set # 2 and performs processing thereof are provided separately. The processing corresponding to S16 to S19 or S20 may be performed as it is without generating the thread SL2. Alternatively, a predetermined number of threads SL1 may be generated, records may be read from the input data set # 1 by the threads SL1, and the processing may be performed. Furthermore, when it is determined in S15 that there are more records, SL1 may be newly generated, and processing corresponding to S10 to S15 may be performed.

FIG. 4B shows a flow of task execution processing according to the second embodiment.

This task execution process indicates a process executed in a task executed in the stage process in the job execution model shown in FIG. 1B.

The processor 101 of the computation node 100 executes the processing of steps S21 to S26 by executing one thread SLd ₁ for reading the record of the input data set #d ₁ and executing the processing. Here, in the task execution process for executing the stage # 1 process of FIG. 1B, the input data set #d ₁ is the input data set # 1, the input data set #d ₂ is the input data set # 2 . Further, in the task execution process for executing the stage # 4 process of FIG. 1B, the input data set #d ₁ is the input data set # 5, the input data set #d ₂ is an input data set # 6.

In S21, the processor 101 acquires one record from the input data set #d _1. Here, the input data set #d ₁ may be stored in the storage 104 of its own calculation node 100 or may be stored in the storage 104 of another calculation node 100. Note that the record acquisition process of acquiring records from the input data set #d ₁ will be described later.

In S22, the processor 101, based on the format 110m corresponding to the record in the input data set #d _1, with respect to interpret the contents of each item contained in the acquired record, the acquired records are included in the condition 110n by applying the criteria for records in the input data set #d _1, in the record to determine whether to satisfy the condition, if necessary, the process proceeds to S23, if not necessary, although not shown S26 move on. In S22, a part of the condition 110n may be applied. If the condition 110n is not defined, the process may proceed to S23 as it is.

In S23, the processor 101, the acquired record, it is determined whether there is a reference to the input data set #d _2. If the result of this determination is affirmative, S24 is performed, while if the result of this determination is negative, S26 is performed.

In S24, the processor 101, for one reference to the input data set #d ₂ of the acquired record, it generates a thread SLd ₂ for getting to process records from the input data set #d _2.

In S25, the processor 101, the acquired record, it is determined whether there are more references to input data set #d _2. If the result of this determination is affirmative, S24 is performed, while if the result of this determination is negative, S26 is performed. Thus, if there are a plurality of references to the input data set #d ₂ in the acquired record, a thread SLd ₂ corresponding to the number of references is generated in S24. Note that when the resources necessary for generating the thread are insufficient, the generation of the thread SLd ₂ may be temporarily suspended. At this time, it may generate one thread SLd ₂ for each of the one reference, may generate one thread SLd ₂ for each of the plural reference (e.g., a predetermined number).

In S26, the processor 101 further determines whether there is another record in the input data set #d _1. If the result is affirmative in this determination, while performing S21, if is negative result of this determination, we end this process, and terminates the thread SLd ₁ executing this process.

On the other hand, the thread SLd _k generated from the thread SLd _k-1 in S24 and S31 to be described later is executed by the processor 101. Here, k represents a natural number of 2 or more. For example, when k = 2, SLd _k-1 represents SLd ₁ , SLd _k represents SLd ₂ , and SLd _{k + 1} represents SLd ₃ . The processor 101 executes the processing of steps S27 to S35 by executing the thread SLd _k . In this embodiment, the processor 101 can execute a plurality of threads (thread SLd ₁ , thread SLd _k, etc.) in parallel. The computing node 100 includes a plurality of processors 101, and a thread generated by one processor 101 may be executed by another processor 101. The number of threads that can be executed in parallel is limited by the resources of the computing node 100 and the like.

In S27, the processor 101, based on the reference method 110o for referring to the input data set #SLd _k using the reference and the references of record input data set #SLd _k-1, the input data set #SLd _k Get a record.

In S28, the processor 101 interprets the content of each item included in the acquired record based on the format 110f (format #d _k ), and the input data set #d included in the condition 110n for the acquired record. By applying the condition for the record of _k , it is determined whether or not the record is necessary. If necessary, the process proceeds to S29. If not necessary, the process proceeds to S35 although not shown. In S28, a part of the condition 110n may be applied. If the condition 110n is not defined, the process may proceed to S29 as it is.

In S29, the processor 101 determines whether or not access to the input data set #d _{k + 1} is further required. If the result of this determination is affirmative, S30 is performed, while if the result of this determination is negative, the process proceeds to S33. For example, in the stage # 1 process of FIG. 1B, a record is acquired from the input record # 1, a record is acquired from the input record # 2 according to the reference in the record, and a record is input from the input record # 3 according to the reference in the record In addition, it is necessary to acquire a record from the input record # 4 according to the reference in the record. Therefore, in S29 executed by the thread SLd _k (SLd ₂ ) generated at k = 2, this determination is affirmative because it requires further access to the input data set # 3, and is generated at k = 3. In S29 executed by the thread SLd _k (SLd ₃ ), since the access to the input data set # 4 is further required, this determination is affirmative, and the thread SLd _k (SLd ₄ ) generated at k = 4. In S29 executed at step S29, this determination is negative because no further access to the input data set is required.

In S30, the processor 101 determines whether or not the acquired record has a reference to the input data set #d _{k + 1} . If the result of this determination is affirmative, S31 is performed, while if the result of this determination is negative, S35 is performed.

In S31, the processor 101 generates a thread SLd _{k + 1} for performing processing by acquiring a record from the input data set #d _{k + 1} for one reference to the input data set #d _{k + 1} of the acquired record.

In S32, the processor 101 determines whether or not the acquired record further includes a reference to the input data set #d _{k + 1} . If the result of this determination is affirmative, S31 is performed, while if the result of this determination is negative, S35 is performed. Thereby, if there are a plurality of references to the input data set #d _{k + 1} in the acquired record, a thread SLd _{k + 1} corresponding to the number of references is generated in S31. Note that when the resources necessary for generating the thread are insufficient, the generation of the thread SLd _{k + 1} may be temporarily suspended. At this time, it may generate one thread SLd k + ₁ for each of the one reference, may generate one thread SLd k + ₁ for each of the plural reference (e.g., a predetermined number).

In S33, the processor s101 builds (generates) a record in a predetermined format based on the acquired record and the build method 110p.

In S <b> 34, the processor 101 executes predetermined processing by applying the stage operation 110 j to the built record, and outputs the execution result. At this time, instead of executing S34 after executing S33 in the thread SLd _k , the processor 101 temporarily stores the built record in the calculation queue of the main memory 105, acquires one record from the calculation queue, and then executes S34. , And applying the stage operation 110j to the record, a predetermined process may be executed on the record, and the execution result may be output. At this time, the processor 101 may execute S34 in a thread different from SLd _k . There may be one or more threads that execute S34. The stage calculation 110j may be applied by acquiring a plurality of records at once from the calculation queue.

In S35, CPU 101 further determines whether there is another record in the range indicated by the reference input dataset #d _k. If this determination result is affirmative, while performing S27, if the result of this determination is negative, then the processing ends, it stops the thread SLd _k executing this process.

In FIG. 4B, read records from the input data set _{#d k-1,} a thread _{SLd k-1} to perform the processing to read records from the input data set #d _k, a thread SLd _k to perform the process separately Although provided, the thread SLd _k-1 may perform the processing corresponding to S27 to S35 without generating the thread SLd _k . Alternatively, a predetermined number of threads SLd ₁ may be generated, and records may be read from the input data set # 1 by the threads SLd ₁ and processed. Furthermore, when it is determined in S26 or S35 that there are more records, SLd ₁ or SLd _k is newly generated, and processing corresponding to S21 to S26 or processing corresponding to S27 to S35 is performed. Good.

FIG. 5A is a diagram for explaining the input data and the process for the input data according to the first embodiment.

Input data set # 2 stores one or more records including “Date & time”, “User”, “Product”, and “Comment”.

In the input data set # 1, one or more records having items of “Product” and “Reference” are collected and managed for each year. That is, the input data set # 1 is divided by year and month. For example, the supervisor process performs parallel data processing by assigning a map task or the like for each of the collected parts (divided parts). At this time, one map task may be responsible for a plurality of divided portions.

“Product” stores a value of a key item (“Product”) used to search for a record in the input data set # 2. “Reference” includes the physical record of the record corresponding to the record in the input data set # 2 that stores the same value as the value of “Product” in the record (reference destination record). A reference indicating the correct storage location is stored. In addition, when there are a plurality of reference destination records in the input data set # 2, “Reference” stores references to the plurality of reference destination records. Note that the input record # 2 may have a structure (for example, a B-tree) in which a record can be searched with a certain key, and may be a reference for storing the value of the key in “Reference”. A plurality of records may correspond to a certain reference.

The record format of input data set # 1 is described in format # 1.

Also, the record format of input data set # 2 is described in format # 2. Further, a method of referring to the record of the input data set # 2 using “Reference” of the input data set # 1 is described in the reference method # 1.

In the map process in the job execution model shown in FIG. 1A, the processor 101 reads the record of the input data set # 1 by executing the map function 121, and grasps the product and the reference in the record by the format # 1. . Then, the map function 121 determines whether or not a predetermined condition is met based on the value of Product, and identifies a reference of a record that satisfies the condition. Then, the processor 101 executes the map function 121 to acquire a record of the input data set # 2 based on the reference method # 1 and the reference.

FIG. 5B is a diagram for explaining the format and reference according to the first embodiment.

Format # 1 is information relating to the record format of the input data set # 1, and in this embodiment, a procedure for interpreting the record of the input data set # 1 is described. Format # 1 includes interpreting an input record (that is, record of input data set # 1) in binary format, interpreting each column of the input record as Text type, Long type, Int type, Int type, It describes that a column of 1 (0) is used as a search key. Here, the column type is indicated using a type declaration in the Java (registered trademark) language, but the present invention is not limited to this.

Format # 2 is information relating to the record format of the input data set # 2, and in this embodiment, a procedure for interpreting the record of the input data set # 2 is described. Format # 2 describes that an input record (that is, a record of input data set # 2) is interpreted in a text format (character string format). The delimiter between the columns in the record is a comma, the first (0) column is the DateTime type, named “Date & Time”, and the second (1) column is the Text type. , “User”, the third (2) column is Text type, “Product”, the fourth (3) column is Text type, “Comment”, based on these To interpret the input column.

The reference method # 1 is a procedure for a method of referring to the record of the input data set # 2 by using “Reference” of the input data set # 1, and the second method of the input record (record corresponding to the format # 1). It is described that a record is obtained by referring to a physical reference using a column as an offset, a third column as a length, and a fourth column as a node ID. Here, the physical reference means that the specified offset (address) of the storage managed by the specified node ID is a starting point and a byte string for the specified length is used as a reference destination record.

FIG. 5C is an example of a schematic diagram illustrating thread generation and thread execution. The upper side of FIG. 5C shows an example in which a process is executed by a single thread, and the lower side of FIG. 5C dynamically generates a thread according to the first embodiment and executes a plurality of threads in parallel. An example is shown. Note that the notation in FIG. 5C follows the following rules.
(*) The horizontal axis represents time.
(*) A round rectangle with a long side in the figure means a series of processing by one thread. The left end of the rounded rectangle represents the time when processing by the thread is started, and the right end of the rounded rectangle represents the time when processing by the thread is ended.
(*) The value inside the rounded rectangle represents information (for example, the value of the first column of the record) indicating the record that is read along with the processing corresponding to the thread.

Here, FIG. 5C shows an example in which each record of the input data set # 1 corresponding to February 2012 (2012-Feb) shown in FIG. 5A is acquired and the process is executed.

As shown in the upper diagram of FIG. 5C, when executed by a single thread, the processor records the second record from the top (2012-Feb) of input data set # 1 corresponding to February 2012 (2012-Feb). Read “Product” as “AX Skirt” and refer to the 7th record (“2012-Feb-07...”) Of input data set # 2 based on the “Reference” value of this record. , Read the third record from the top of input data set # 1 (the value of “Product” is “BB Book”), and based on the value of “Reference” of this record, the eighth record of input data set # 2 ("2012-Feb-08 ..."), and based on the "Reference" value, Referring to the tenth record (“2012-Feb-08...”) Of the tuset # 2, and further based on the value of “Reference”, the eleventh record (“2012-Feb” of the input data set # 2) -08...))) And process. That is, the reading of the record from the storage and the processing for it are performed sequentially.

On the other hand, in the present embodiment, as shown in the lower diagram of FIG. 5C, the map function 121 first corresponds to February 2012 (2012-Feb) of the input data set # 1 by the thread 5a. Read the second record (“Product” is “AX Skirt”), and based on the “Reference” value of this record, the seventh record “2012-Feb-07... )) Is generated and executed.

Next, the map function 121 reads the third record from the top of the input data set # 1 corresponding to 2012-Feb (the value of “Product” is “AX Skirt”) by the thread 5a, and the “Reference” of this record is read. Based on the four values, the thread 5c for referring to the eighth record (“2012-Feb-08...”) Of the input data set # 2 and the tenth record (“2012” of the input data set # 2) -Feb-08 ... ") thread 5d for referencing, 11th (" 2012-Feb-08 ... ") thread 5e for referring to the input data set # 2, and input data Thread 5 for referring to the twelfth record of set # 2 ("2012-Feb-09 ...") Sequentially generates and executed.

Next, the map function 121 reads the fourth record from the top (the value of “Product” is “BC Bike”) corresponding to the 2012-Feb of the input data set # 1 by the thread 5a, and the “Reference” of this record is read. Based on the two values, a thread 5g for referring to the sixth record of the input data set # 2 and a thread 5h for referring to the ninth record of the input data set # 2 are generated and executed.

Next, the map function 121 reads the fifth record from the top of the input data set # 1 corresponding to 2012-Feb (the value of “Product” is “BD Flower”) by the thread 5a, and the “Reference” of this record is read. Based on one value, a thread 5i for referring to the fifth record (“2012-Feb-03...”) Of the input data set # 2 is generated and executed.

As shown in the lower diagram of FIG. 5C, when a thread is dynamically generated, a record is read and processed in the thread, and a plurality of threads are executed in parallel to execute a single thread. Compared to, the execution time of the process can be shortened. FIG. 5D is a first diagram for explaining the input data and the process for the input data according to the second embodiment. FIG. 5E is a second diagram for explaining the input data and the process for the input data according to the second embodiment.

In the job execution plan according to the second embodiment, the record of the input data set # 3 is further referred to by using the value of the record of the input data set # 2. Therefore, as shown in FIG. 5D, the application 110 further includes a reference method # 2 for referring to the input data set # 3 based on the record value (“User” value) of the input data set # 2. Provided. Here, the value of “User” is not a reference indicating a physical position of a record of the input data set # 3 corresponding to the value, but a reference indicating a logical position (searchable by the value). is there.

FIG. 5E shows an input data set # 3 and an input data set # 4.

The input data set # 4 stores one or more records including “User”, “Gender”, “Zip”, and “Address”.

In the input data set # 3, one or more records having items of “User” and “Reference” are collected and managed for each predetermined range. In “User”, the value of an item which is a key used for searching for a record in the input data set # 4 is stored. In “Reference”, a reference indicating a physical storage position of a record (reference destination record) storing the same value as the value of “Product” in the record in the input data set # 4 is stored. When there are a plurality of reference destination records in the input data set # 3, references to the plurality of reference destination records are stored. Note that the input record # 3 may have a structure (for example, a B-tree) that allows a record to be searched with a certain key, and may be a reference for storing the value of the key in “User”. A plurality of records may correspond to a certain reference.

The record format of input data set # 3 is described in format # 3. The record format of the input data set # 4 is described in format # 4. Further, a method of referring to the record of the input data set # 4 using “Reference” of the input data set # 3 is described in the reference method # 3. A procedure for generating a record to be output subsequently is described in the build method based on the acquired record.

In the stage # 1 process in the job execution model shown in FIG. 1B, the processor 101 reads the record of the input data set # 1 by executing the generalized stage function 124, and “Product” in the record by the format # 1 , “Reference”. Then, the generalization stage function 124 determines whether or not a predetermined condition is met based on the value of “Product”, and specifies “Reference” of a record that satisfies the condition. Then, the generalized stage function 124 acquires a record of the input data set # 2 based on the reference method # 1 and “Reference”.

Next, the processor 101 executes the generalized stage function 124 to grasp “User”, “Product”, and “Comment” of the record of the read input data set # 2 according to the format # 2. Then, the generalization stage function 124 acquires a record of the input data set # 3 based on the value of “User” and the reference method # 2.

By executing the generalized stage function 124, the processor 101 grasps “Reference” in the record of the acquired input data set # 3 according to the format # 3. Then, the generalization stage 124 acquires a record of the input data set # 4 based on the reference method # 3 and “Reference”.

Next, the processor 101 executes the generalized stage function 124 to grasp “User”, “Gender”, “Zip”, and “Address” of the record of the read input data set # 4 according to the format # 4. Then, the processor 101 executes the generalized stage function 124 to construct and output a record including “User”, “Product”, “Comment”, “Gender”, and “Zip” based on the build method. .

FIG. 5F is a diagram for explaining a format according to the second embodiment.

Format # 3 is information related to the record format of the input data set # 3. In this embodiment, a procedure for interpreting the record of the input data set # 3 is described. Format # 3 includes interpreting the input record (that is, record of input data set # 3) in binary format, interpreting each column of the input record as Text type, Long type, Int type, Int type, It describes that a column of 1 (0) is used as a search key.

Format # 4 is information related to the record format of the input data set # 4. In this embodiment, a procedure for interpreting the record of the input data set # 4 is described. Format # 4 describes that an input record (that is, a record of input data set # 4) is interpreted in a text format (character string format). The delimiter between the columns in the record is a comma, the first (0) column is Text type, named “User”, the second (1) column is Text type, Named “Gender”, the third (2) column is Text type, named “Zip”, the fourth (3) column is Text type, named “Address” and entered based on these Describes interpreting columns.

FIG. 5G is a diagram for explaining the reference method according to the second embodiment.

The reference method # 2 is a procedure for referring to the record of the input data set # 3 using the value of the record of the input data set # 2, and the second method of the input record (record corresponding to the format # 2). It describes that a column is used as a reference key and a record is obtained by referring to it with a logical reference. Here, the logical reference means that the reference destination data set is searched by the designated key value to identify the reference destination record.

The reference method # 3 is a procedure for a method of referring to the record of the input data set # 6 by using “Reference” of the input data set # 4, and the second method of the input record (record corresponding to the format # 5). It is described that a record is obtained by referring to a physical reference using a column as an offset, a third column as a length, and a fourth column as a node ID. Here, the physical reference means that the specified offset (address) of the storage managed by the specified node ID is a starting point and a byte string for the specified length is used as a reference destination record.

FIG. 5H shows an example of a catalog according to the second embodiment.

In this embodiment, the format and the reference method are described in, for example, program code. Therefore, when the user prepares the format and the reference method, the user needs to be able to create the program code. However, not all users can create program codes. Therefore, in this embodiment, the user can specify a part of job information such as a format and a reference method by describing a catalog that is easier than the program code in the application, and based on the catalog, the parallel processing is performed. Let the data processing system execute data processing jobs. At this time, the parallel data processing system may execute the job after converting the catalog into a format or a reference method, or may directly execute the job using the catalog.

The catalog shown in FIG. 5H is described in XML (Extensible Markup Language), for example, and includes description sections 50a to 50d related to four data structures.

In the description part 50a, the data set of “user_comment” corresponding to the input data set # 2 has a format in which the text is divided into columns, the first (0) column is interpreted in the DateTime type, and this is used as a split key. It is described that the delimiter between columns is a comma.

In addition, in the description unit 50b, the data set of “user_comment.product.index” corresponding to the input data set # 1 is in a format as a local secondary index corresponding to “user_comment”, and is in “user_comment”. On the basis of this, it is described that the third (2) column when the column delimiter is a comma is interpreted as a Text type and used as an index key. The

description parts

50c and 50d have the same description as the above description part. Note that it is not necessary to explicitly specify all necessary job information in the catalog. For those that are not explicitly specified, the system module etc. performs parallel data processing in accordance with the specified rules. Also good. For example, in this example, no split key is explicitly specified for “user_comment.product.index”, but “user_comment.product.index” is defined as a local secondary index. The same division as the “user_comment” that is the basis is performed, that is, “user_comment.product.index” as a secondary index may be configured for each division part of “user_comment”.

FIG. 6A is an example of a schematic diagram for explaining record acquisition according to the first embodiment. 6A to 9C are descriptions of the first embodiment, but are not limited to the first embodiment but are also applied to the second embodiment.

When the processor 101 of the computation node 100 executes a thread and obtains a record using the thread, if the record is a record stored in the storage 104 of its own computation node 100, the storage 104 Get a record. On the other hand, if the record is stored in the storage 104 of another calculation node 100, a record acquisition request for acquiring the record from the calculation node 100 to the other calculation node 100, for example, via the local area network 200. The calculation node 100 acquires a record by receiving the record acquired from the storage 104 by the other calculation node 100 in accordance with the record acquisition request. At this time, a session is established between the computation node 100 and another computation node 100. When a plurality of threads are executed, a session is created for each record acquisition request generated for each thread. In this case, as the number of threads increases, the number of extended sessions increases, and processing related to session management and control increases, resulting in a decrease in efficiency.

In contrast, as shown below, a session may be created for each block in which a plurality of record acquisition requests are collected.

FIG. 6B is an example of a schematic diagram for explaining blocking at the time of data acquisition according to the second embodiment.

The processor 101 of the calculation node 100 collects a plurality of record acquisition requests generated by a plurality of threads into one block (blocked record acquisition request), and the calculation node 100 and other calculation nodes 100 in units of the block. Create a session with Thereby, the number of sessions spanned between the computation nodes 100 can be reduced, and a reduction in processing efficiency can be prevented.

FIG. 7A shows a flow of record acquisition processing in a calculation node that acquires records according to the first embodiment.

This record acquisition processing corresponds to the processing of S10 and S16 in FIG. 4A and S21 and S27 in FIG. 4B.

In S41, the data reader / writer 140 determines whether or not the reading (acquisition) of the record instructed from the system module 120 is the acquisition of the record from the storage 104 of its own computation node 100, that is, the acquisition of the local record. to decide. If the result of this determination is affirmative, S42 is performed, while if the result of this determination is negative, S44 is performed.

In S42, the data read / write device 140 causes the storage manager 160 to issue a data read request for reading data necessary for acquiring a record to the storage 104 via the OS 170 and the HBA 103. Specifically, the storage manager 160 stores data read request information in the data read request management table 700 (see FIG. 7C), and sends a data read request to the request queue 740 in the main memory 105 of the computing node 100. Add The OS 170 acquires a data read request from the request queue 740 and issues it to the storage 104 by the HBA 103.

In S43, the data reader / writer 140 places its own thread in a suspended state and ends the process.

In S44, the data reader / writer 140 causes the network manager 150 to transmit a record acquisition request message to the other calculation nodes 100 via the OS 170 and the NIC 102. Specifically, the network manager 150 stores the information of the record acquisition request message in the remote record acquisition request management table 710 (see FIG. 7C) and records it in the transmission queue 720 of the main memory 105 of the computing node 100. Add an acquisition request message.

In S45, the data reader / writer 140 puts its own thread in a suspended state and ends the process.

On the other hand, the storage 104 of the computing node 100 receives the data read request issued from the request queue 740, reads the data corresponding to the data read request, and transmits it to the HBA 103. The OS 170 adds the data read from the HBA 103 to the completion queue 750 in the main memory 105.

Thereafter, in S46, the data reader / writer 140 acquires data corresponding to the data read request from the completion queue 750, and extracts a record from the data. record

In S47, the data reader / writer 140 identifies a thread that uses the received record based on the data read request management table 700, restarts the thread, and ends the record acquisition process.

On the other hand, the record acquisition request message stored in the transmission queue 720 is transmitted by the NIC 102 to the calculation node 100 of the transmission destination. In addition, an acquisition completion message for a record acquisition request message transmitted from another calculation node 100 is stored in the reception queue 730 by the NIC 102.

In S48, the network manager 150 extracts a record from the acquisition completion message in the reception queue 730 and passes it to the system module 120.

In S49, the data reader / writer 140 identifies a thread that uses the received record based on the remote record acquisition request management table 710, restarts the thread, and ends the record acquisition process.

FIG. 7B shows the flow of a record acquisition process in a calculation node that manages records according to the first embodiment.

In the calculation node 100 to which the record acquisition request message is transmitted, the NIC 102 acquires the record acquisition request message, and the OS 170 stores it in the reception queue 760 of the main memory 105.

In S 50, the network manager 150 of the calculation node 100 of the transmission destination acquires a record acquisition request message from the reception queue 760 and passes it to the data reader / writer 140.

In S51, based on the record acquisition request message, the data read / write device 140 uses the storage manager 160 to read data necessary for acquisition of records to the storage 104 via the OS 170 and the HBA 103. Is issued and the process is terminated. Specifically, the storage manager 160 adds a data read request to the request queue 780 in the main memory 105 of the computing node 100. The OS 170 acquires a data read request from the request queue 740 by the HBA 103 and issues it to the storage 104.

The storage 104 receives the data read request issued from the request queue 780, reads the record corresponding to the data read request, and transmits it to the HBA 103. The OS 170 adds the record read by the HBA 103 to the completion queue 790 in the main memory 105.

Thereafter, in S52, the data reader / writer 140 extracts a record from the data corresponding to the data read request from the completion queue 790.

In S53, the data reader / writer 140 includes the received record in the acquisition completion message, and the network manager 150 transmits the acquisition completion message to the calculation node 100 that is the transmission source of the record acquisition request message, and ends the processing. Specifically, the network manager 150 adds an acquisition completion message to the transmission queue 760 in the main memory 105 of the computation node 100.

The acquisition completion message stored in the transmission queue 760 is transmitted by the NIC 102 to the calculation node 100 that is the transmission source of the record acquisition request message. This acquisition completion message is stored in the reception queue 730 by the NIC 102 in the calculation node 100 of the transmission source.

The process illustrated in FIG. 7A may be executed in threads (SL1, SL2, SLd ₁ , SLd ₂ , SLd _k , SLd _{k + 1,} etc.) that perform the processes of S10 and S16 in FIG. 4A and S21 and S27 in FIG. 4B. . For example, the CPU 101 may execute the procedure corresponding to S41 to S45 in the same thread SL1 by executing the data acquisition process of S10 in the thread SL1. The process illustrated in FIG. 7A is different from the threads (SL1, SL2, SLd ₁ , SLd ₂ , SLd _k , SLd _{k + 1,} etc.) that perform the processes of S10 and S16 in FIG. 4A and S21 and S27 in FIG. 4B. It may be executed in a thread. For example, the CPU 101 may execute the procedure corresponding to S41 to S45 in another thread by executing the data acquisition process of S10 in the thread SL1. In this case, separate threads may be provided for the procedures corresponding to S41 to S45, the procedures corresponding to S46 to S47, and the procedures corresponding to S48 to S49, or the same thread may be used. . A plurality of threads may be provided for each procedure. Further, another thread may be provided to drive a procedure corresponding to S46 to S47 and a procedure corresponding to S48 to S49. These driving operations may be performed by means such as an interrupt or a signal generated when the HBA 103 and the NIC 102 add the completion queue 750 and the reception queue 730.

The processing illustrated in FIG. 7B may be provided in separate threads or in the same thread in addition to the procedures corresponding to S50 to S51 and the procedures corresponding to S52 to S53. A plurality of threads may be provided for each procedure. Further, another thread may be provided to drive the procedure corresponding to S52 to S53. Such driving may be performed by means such as an interrupt or a signal generated by the addition of the HBA 103 to the completion queue 790.

FIG. 7C shows the configuration of the data read request management table and the remote record acquisition request management table according to the first embodiment.

The data read request management table 700 includes a thread ID 701, a request issue time 702, and a data read request 703 as information for each data read request. The data read request 703 includes a device ID 704, an offset address 705, a read length 706, and a buffer address 707. Various information is as follows.
(*) The thread ID 701 indicates an ID for identifying the thread that issued the data read request.
(*) Request issue time 702 indicates the time at which a data read request is issued.
(*) Data read request 703 indicates the content of the data read request.
(*) The device ID 704 indicates the device ID of the storage 104 that is the transmission destination of the data read request.
(*) An offset address 705 indicates an address (offset address) in the storage 104 in which the record of the data read request target is stored.
(*) Read length 706 indicates the data length (byte length) of the record to be read.
(*) A buffer address 707 indicates an address of an area (buffer) on the main memory 105 that stores a record to which a data read request is made.

The remote record acquisition request management table 710 includes a thread ID 711, a request issue time 712, and a record acquisition request 713 as information for each record acquisition request. The record acquisition request 713 has a calculation node ID 714, a record reference 715, and a buffer address 716. Various information is as follows.
(*) The thread ID 711 indicates an ID for identifying the thread that issued the record acquisition request.
(*) The request issue time 712 indicates the time when the record acquisition request is issued.
(*) Record acquisition request 713 indicates the contents of the record acquisition request.
(*) The calculation node ID 714 indicates the ID (calculation node ID) of the calculation node that is the transmission destination of the record acquisition request.
(*) Record reference 715 indicates reference information to a record that is a target of a record acquisition request.
(*) The buffer address 716 indicates the address of an area (buffer) on the main memory 105 that stores the record of the data read request target.

Next, a record acquisition process according to a modification of the first embodiment will be described.

FIG. 7D shows the flow of a record acquisition process in a calculation node that acquires a record according to a modification of the first embodiment.

In S61, the data reader / writer 140 determines whether or not the reading (acquisition) of the record instructed from the system module 120 is the acquisition of the record from the storage 104 of its own computation node 100, that is, the acquisition of the local record. to decide. If the result of this determination is affirmative, S62 is performed, while if the result of this determination is negative, S64 is performed.

In S62, the data reader / writer 140 causes the storage manager 160 to issue a data read request to the storage 140. The specific process is the same as S42 in FIG. 7A.

In S63, the data read / write device 140 puts its own thread in a suspended state and ends the process.

In S64, the data reader / writer 140 sends a record acquisition request message to each computation node 100 in the blocked

queues

800, 810, and 820 provided corresponding to each computation node 100 in the main memory 105. The record acquisition request message is added to the blocked queue corresponding to the calculation node 100 of the transmission destination, and the record acquisition request message is added to the blocked remote record acquisition request management table 830. Here, in the blocked remode record acquisition request management table 830, a plurality of record acquisition request messages having the same calculation node 100 as a transmission destination are combined into one request (blocked remote record acquisition request).

In S65, the data reader / writer 140 places its own thread in a suspended state and ends the process.

In S66, the network manager 150 extracts a blocked remote record acquisition request including a plurality of record acquisition request messages from each of the blocked

queues

800, 810, and 820.

In S67, the network manager 150 transmits a blocked remote record acquisition request message via the OS 170 and the NIC 102, and ends the process. Specifically, the network manager 150 stores the blocked remote record acquisition request message in the transmission queue 840 of the main memory 105, and the NIC 102 sends the blocked remote record acquisition request message from the transmission queue 840 to the destination calculation node 100. Send. Thus, since a plurality of record acquisition requests are made into one blocked remote record acquisition request message, the number of sessions established during communication can be reduced.

Thereafter, the OS 170 receives the blocked record acquisition completion message transmitted from the other calculation node 100 by the NIC 102 and stores it in the reception queue 850.

In S68, the network manager 150 acquires a blocked record acquisition completion message from the reception queue 850, extracts a plurality of records from the blocked record acquisition completion message, and passes them to the system module 120.

In S69, the data reader / writer 140 identifies a thread that uses the received record based on the blocked remote record acquisition request management table 830, restarts the thread, and ends the record acquisition process.

FIG. 7E shows the flow of record acquisition processing in a calculation node that manages records according to a modification of the first embodiment.

In the calculation node 100 that is the transmission destination of the blocked remote record acquisition request message, the NIC 102 acquires the blocked remote record acquisition request message and stores it in the reception queue 860 of the main memory 105.

In S70, the network manager 150 of the destination computation node 100 acquires the blocked remote record acquisition request message from the reception queue 860, extracts a plurality of record acquisition requests from the blocked remote record acquisition request message, and reads and writes the data read / write device 140.

In S <b> 71, the data reader / writer 140 uses the storage manager 160 to read data necessary for acquiring a plurality of records to the storage 104 via the HBA 103 based on the plurality of record acquisition request messages. The data read request is issued, and the process ends. Specifically, the storage manager 160 adds a plurality of data read requests to the request queue 880 in the main memory 105 of the computing node 100. The HBA 104 acquires a data read request from the request queue 780 and issues it to the storage 104.

The storage 104 receives the data read request issued from the request queue 880, reads the record corresponding to the data read request, sends it to the HBA 103, and sends the data read by the HBA 103 to the completion queue 890 of the main memory 105. to add.

Thereafter, in S72, the storage manager 160 acquires a plurality of records corresponding to the data read request from the completion queue 890, extracts the records from the data, and passes them to the system module 120.

In S73, the data reader / writer 140 includes the received plurality of records in the blocked acquisition completion message, and the network manager 150 sends a blocked acquisition completion message to the computing node 100 that is the transmission source of the blocked remote record acquisition request message. Is transmitted, and the process ends. Specifically, the network manager 150 adds a blocked acquisition completion message to the transmission queue 870 in the main memory 105 of the computing node 100. The blocked acquisition completion message stored in the transmission queue 870 is transmitted by the NIC 102 to the calculation node 100 that is the transmission source of the blocked remote record acquisition request message. This blocked acquisition completion message is stored in the reception queue 850 by the NIC 102 in the calculation node 100 of the transmission source.

The process illustrated in FIG. 7D may be executed in threads (SL1, SL2, SLd ₁ , SLd ₂ , SLd _k , SLd _{k + 1,} etc.) that perform the processes of S10 and S16 in FIG. 4A and S21 and S27 in FIG. 4B. . For example, the CPU 101 may execute the procedure corresponding to S61 to S65 in the same thread SL1 by executing the data acquisition process of S10 in the thread SL1. The process illustrated in FIG. 7D is different from the threads (SL1, SL2, SLd ₁ , SLd ₂ , SLd _k , SLd _{k + 1,} etc.) that perform the processes of S10 and S16 in FIG. 4A and S21 and S27 in FIG. 4B. It may be executed in a thread. For example, the CPU 101 may execute the procedure corresponding to S61 to S65 in another thread by executing the data acquisition process of S10 in the thread SL1. In this case, separate threads may be provided for the procedures corresponding to S61 to S65, the procedures corresponding to S66 to S67, and the procedures corresponding to S68 to S69, or the same thread may be used. . A plurality of threads may be provided for each procedure. Further, another thread may be provided to drive the procedure corresponding to S66 to S67 and the procedure corresponding to S68 to S69. These driving operations may be performed by means such as an interrupt or a signal generated when the HBA 103 and the NIC 102 add the completion queue 750 and the reception queue 730.

The processing illustrated in FIG. 7E may be performed in a separate thread, or may be performed in the same thread, separately from the procedure corresponding to S70 to S71 and the procedure corresponding to S72 to S73. A plurality of threads may be provided for each procedure. For example, a plurality of threads may be generated according to the blocked record acquisition request message acquired in S70, and the data read request may be executed by executing S71 in each of the threads. At this time, the same number of threads as the number of record acquisition requests included in the message may be generated, or a predetermined number of threads may be generated. In order to drive the procedure corresponding to S72 to S73, another thread may be provided. Such driving may be performed by means such as an interrupt or a signal generated by the addition of the HBA 103 to the completion queue 790.

FIG. 7F shows the configuration of a blocked remote record acquisition request management table according to a modification.

The blocked remote record acquisition request management table 830 includes a request issue time 832, the number of requests 833, and one or more record acquisition requests 834 as information for each blocked remote record acquisition request. The record acquisition request 834 includes a thread ID 831, a calculation node ID 835, a record reference 836, a buffer address 837, and a completion flag 838. Various information is as follows.
(*) Request issue time 832 indicates the time at which the blocked remote record acquisition request was issued.
(*) The number of requests 833 indicates the number of record acquisition requests included in the bricked remote record acquisition request.
(*) Record acquisition request 834 indicates the contents of the record acquisition request.
(*) The thread ID 831 indicates an ID for identifying the thread that issued the record acquisition request.
(*) The calculation node ID 835 indicates the ID (calculation node ID) of the calculation node that is the transmission destination of the record acquisition request.
(*) Record reference 836 indicates reference information to the record that is the target of the record acquisition request.
(*) The buffer address 837 indicates the address of an area (buffer) on the main memory 105 that stores the record that is the target of the record acquisition request.
(*) The completion flag 838 is a flag indicating whether or not a record corresponding to the record acquisition request has been acquired.

In the above description, the procedure for acquiring a record in a job for performing parallel data processing has been described focusing on the characteristic points of the present invention. However, various implementation forms are conceivable for the procedure.

In order to extract records from a data set, information on how records are arranged in the data set is indispensable. For example, when a text file is used as a data set and one of the lines is handled as a record, the record and the record are separated by a line feed code. In order to extract a record from the text file, information that the record is delimited by a line feed code is indispensable. As another example, when a data set has a structure such as a B-tree, some records may be packed in an appropriate data structure in a page which is a unit of access on the storage. In order to extract a record from such a structure, information on the structure (page length, page header / footer structure, record header / footer structure in a page, etc.) is indispensable. Furthermore, when the data set is compressed or encrypted, the procedure information for decryption is indispensable for obtaining a record from the data set. Information regarding the configuration of the data set for acquiring records in this way may be defined by the application as part of the job information, similarly to the format and the like. Alternatively, the application does not have to explicitly specify all of the information related to the configuration of the data set. As for the configuration of the data set that is not defined by the application, the job may be executed assuming that the system module or the like has a predetermined configuration. At this time, the system module or the like may determine the configuration of the data set based on information about the data set or the data set. For example, if the application does not explicitly define the structure of the data set, but the data set is determined to be a text file, the job may be executed assuming that the records are separated by line feed codes. Good. At this time, information on the configuration of the data set defined by the application may be notified to the data reader / writer via the system module.

When extracting a record from a data set, the system module or the like may cache a part of the data in the data set in the main memory to reduce access to the storage. For example, when acquiring a record by scanning a text file, instead of accessing the storage for each record, the data is read from the text file in units of 1 megabyte at a time and stored in the main memory. A record may be taken out from the inside.

FIG. 8A shows a configuration of a node-level resource constraint management table according to the first embodiment.

8A is managed by a supervisory supervisor process that supervises each computation node 100.

The node level resource constraint management table 900 includes a computation node ID 901 and a resource constraint 902 as information for each computation node. The resource constraint 902 has a thread count 903 and a main memory allocation 904. Various information is as follows.
(*) The calculation node ID 901 indicates the ID of the calculation node.
(*) Resource constraint 902 indicates a resource constraint in the computation node corresponding to the computation node ID 901.
(*) The number of threads 903 indicates the maximum number of threads that can be generated in the calculation node corresponding to the calculation node ID 901.
(*) Main memory allocation 904 indicates the maximum amount of main memory that can be allocated in the calculation node corresponding to the calculation node ID 901.

The node-level resource constraint management table 910 on the lower side of FIG. 8A is managed by a supervisor process in each computation node 100.

The node level resource constraint management table 910 includes a calculation node ID 911, a resource constraint 912, and a resource usage 913 as information of its own calculation node. The resource constraint 912 has a thread number 914 and a main memory allocation 915. The resource use 913 has a thread number 916 and a main memory allocation 917. Various information is as follows.
(*) The calculation node ID 911 indicates the ID (calculation node ID) of its own calculation node.
(*) Resource constraint 912 indicates a resource constraint in its own computation node.
(*) Resource use 913 indicates a resource used in its own computation node.
(*) The number of threads 914 indicates the maximum number of threads that can be generated in its own computation node.
(*) Main memory allocation 915 indicates the maximum amount of main memory that can be allocated in its own computation node.
(*) The number of threads 916 indicates the number of threads actually generated in its own computation node.
(*) Main memory allocation 917 indicates the storage amount of the main memory actually allocated in its own computation node.

FIG. 8B shows the configuration of a job level resource constraint management table according to the first embodiment.

The resource level management table 920 at the upper level of FIG. 8B is managed by the overall supervisor process.

The job level resource constraint management table 920 includes a job ID 921, a calculation node ID 922, and a resource constraint 923 as information for each job. The resource constraint 923 has a thread number 924 and a main memory allocation 925. Various information is as follows.
(*) The job ID 921 indicates a job ID (job ID).
(*) The calculation node ID 922 indicates the ID (calculation node ID) of the calculation node that executes the job with the job ID 921.
(*) Resource constraint 923 indicates a resource constraint for the job with job ID 921 in the computation node with computation node ID 922.
(*) The thread number 924 indicates the maximum number of threads that can be generated for the job with the job ID 921 in the calculation node with the calculation node ID 922.
(*) Main memory allocation 925 indicates the maximum amount of main memory that can be allocated to the job with job ID 921 in the calculation node with calculation node ID 922.

The node-level resource constraint management table 930 on the lower side of FIG. 8B is managed by a supervisor process in each computation node 100.

The node level resource constraint management table 930 includes a job ID 931, a calculation node ID 932, a resource constraint 933, and a resource usage 934 as information for each job in its own computation node. The resource constraint 933 has a thread number 935 and a main memory allocation 936. The resource use 934 has a thread number 937 and a main memory allocation 938. Various information is as follows.
(*) Job ID 931 indicates a job ID (job ID).
(*) The calculation node ID 932 indicates an ID (calculation node ID) of its own calculation node.
(*) Resource constraint 933 indicates a resource constraint for the job with job ID 931 in its own computation node 100.
(*) Resource use 934 indicates the resource used for the job with the job ID 931 in its own computation node 100.
(*) The number of threads 935 indicates the maximum number of threads that can be generated for the job with the job ID 931 in its own computation node 100.
(*) Main memory allocation 936 indicates the maximum amount of main memory that can be allocated to the job with the job ID 931 in its own computation node 100.
(*) The number of threads 937 indicates the number of threads actually generated for the job with the job ID 931 in its own computation node 100.
(*) The main memory allocation 938 indicates the storage amount of the main memory actually allocated to the job with the job ID 931 in its own computation node 100.

FIG. 8C shows a configuration of a process level resource constraint management table in the supervisor process according to the first embodiment.

The process level resource constraint management table 940 is managed by the supervisory supervisor process.

The process level resource constraint management table 940 includes a process ID 941, a job ID 942, a calculation node ID 943, and a resource constraint 944 as information for each process. The resource constraint 944 has a thread number 945 and a main memory allocation 946. Various information is as follows.
(*) Process ID 941 indicates a process ID (process ID).
(*) Job ID 942 indicates a job ID.
(*) The calculation node ID 943 indicates the ID (calculation node ID) of the calculation node that executes the process with the process ID 941 of the job with the job ID 921.
(*) Resource constraint 944 indicates a resource constraint for the process with the process ID 941 of the job with the job ID 942 in the computation node with the computation node ID 943.
(*) The number of threads 945 indicates the maximum number of threads that can be generated for the process with the process ID 941 of the job with the job ID 942 in the calculation node with the calculation node ID 943.
(*) Main memory allocation 946 indicates the maximum storage amount of main memory that can be allocated to the process with the process ID 941 of the job with the job ID 942 in the calculation node with the calculation node ID 943.

FIG. 8D shows a configuration of a process level resource constraint management table in the supervisor process of each node according to the first embodiment.

The process level resource constraint management table 950 is managed by a supervisor process in each computation node 100.

The process level resource constraint management table 950 includes a process ID 951, a job ID 952, a computation node ID 953, a resource constraint 954, and a resource usage 955 as information for each process in each job in its own computation node. The resource constraint 954 has a thread count 956 and a main memory allocation 957. The resource use 955 has a thread count 958 and a main memory allocation 959. Various information is as follows.
(*) Process ID 951 indicates the ID of the process.
(*) Job ID 952 indicates a job ID.
(*) The calculation node ID 953 indicates the ID (calculation node ID) of its own calculation node 100.
(*) Resource constraint 954 indicates a resource constraint for the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.
(*) Resource use 955 indicates the resource used for the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.
(*) The number of threads 956 indicates the maximum number of threads that can be generated for the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.
(*) Main memory allocation 957 indicates the maximum amount of main memory that can be allocated to the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.
(*) The number of threads 958 indicates the number of threads actually generated for the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.
(*) Main memory allocation 959 indicates the storage amount of the main memory actually allocated to the process with the process ID 951 of the job with the job ID 952 in its own computation node 100.

FIG. 8E shows the configuration of a task level resource constraint management table in the overall supervisor process according to the first embodiment.

The task level resource constraint management table 960 is managed by the supervisory supervisor process.

The task level resource constraint management table 960 includes a task ID 961, a process ID 962, a job ID 963, a calculation node ID 964, and a resource constraint 965 as information for each task. The resource constraint 944 has a thread number 945 and a main memory allocation 946. Various information is as follows.
(*) Task ID 961 indicates a task ID (process ID).
(*) Process ID 962 indicates a process ID (process ID).
(*) Job ID 963 indicates a job ID.
(*) The calculation node ID 964 indicates the ID (calculation node ID) of the calculation node that executes the task with the task ID 961 of the process with the process ID 962 of the job with the job ID 963.
(*) Resource constraint 965 indicates a resource constraint for the task with the task ID 961 of the process with the process ID 962 of the job with the job ID 963 in the computation node with the computation node ID 964.
(*) The number of threads 966 indicates the maximum number of threads that can be generated for the task with the task ID 961 of the process with the process ID 962 of the job with the job ID 963 in the calculation node with the calculation node ID 964.
(*) Main memory allocation 967 indicates the maximum storage amount of main memory that can be allocated to the task with task ID 961 of the process with process ID 962 of job ID 963 in the calculation node with calculation node ID 964.

FIG. 8F illustrates a configuration of a task level resource constraint management table in the supervisor process of each node according to the first embodiment.

The task level resource constraint management table 970 is managed by a supervisor process in each computation node 100.

The task level resource constraint management table 970 includes a task ID 971, a process ID 972, a job ID 973, a calculation node ID 974, a resource constraint 975, and a resource usage 976 as information on each task of each process in each job in its own computation node. . The resource constraint 975 has a thread number 977 and a main memory allocation 978. The resource use 976 has a thread number 979 and a main memory allocation 980. Various information is as follows.
(*) Task ID 971 indicates the ID of the task.
(*) Process ID 972 indicates the ID of the process.
(*) Job ID 973 indicates a job ID.
(*) The calculation node ID 974 indicates the ID (calculation node ID) of its own calculation node 100.
(*) The resource constraint 975 indicates a resource constraint for the task with the task ID 971 of the process with the process ID 972 of the job with the job ID 973 in its own computation node 100.
(*) Resource use 976 indicates a resource that is used for the task with the task ID 971 of the process with the process ID 972 of the job with the job ID 973 in its own computation node 100.
(*) The number of threads 977 indicates the maximum number of threads that can be generated for the task with the task ID 971 of the process with the process ID 972 of the job with the job ID 973 in its own computation node 100.
(*) Main memory allocation 978 indicates the maximum amount of main memory that can be allocated to the task of task ID 971 of the process ID 972 of the job of job ID 973 in its own computation node 100.
(*) The number of threads 979 indicates the number of threads actually generated for the task with the task ID 971 of the process with the process ID 972 of the job with the job ID 973 in the own computation node 100.
(*) Main memory allocation 980 indicates the storage amount of main memory that is actually allocated to the task of task ID 971 of the process ID 972 of the job of job ID 973 in its own computation node 100.

FIG. 8G shows the flow of the integrated resource constraint management process according to the first embodiment.

In S81, when the supervisory process in the computing node 100 that supervises the supervisor process of each computing node 100 assigns a new task, it calculates resource constraints for the new task. The resource constraint specified by the user may be used, or the resource constraint may be calculated (for example, proportionally distributed) based on the policy specified by the user.

In S82, the supervisory supervisor process adds a record for the new task to the task level resource constraint management table 960 as shown in FIG. 8E, and stores the resource constraint calculated for the resource constraint of the record. In addition, the supervisory supervisor process selects a computation node 100 that can execute a task within the scope of the resource constraint, and transmits the resource constraint related to the computation node to the computation node 100.

In S83, the supervisory supervisor process assigns a task to the selected computation node 100 and ends the process.

In S84, the supervisor process of the computing node 100 to which the task is assigned receives the resource constraint transmitted in Step S82, and registers the resource constraint in the task level resource constraint management table 970 as shown in FIG. 8F.

In S85, the system module 120 of the compute node 100 to which the task is assigned executes the assigned task.

FIG. 8H illustrates a flow of resource constraint management processing in each computation node according to the first embodiment.

The resource constraint management process is realized, for example, when the system module 120 executes a task. The process at the upper left of FIG. 8H (steps S90 to S94) is a process executed when attempting to generate a thread in a task. This is, for example, processing executed in S13 of FIG. 4A and S24 and S31 of FIG. 4B.

In S90, the system module 120 (specifically, the map function 121) determines whether there are sufficient resources to generate a thread. Here, as to whether or not there are sufficient resources to generate a thread, a task level resource constraint management table 970, a process level resource constraint management table 950, a job level resource constraint management table 930, and a node level Each resource constraint management table 910 can be referred to, and it can be determined whether or not the resources available within the resource constraint range at each level are more than the resources sufficient to generate threads.

If the result of this determination is affirmative, S91 is performed. If the result of this determination is negative, S93 is performed.

In S91, the system module 120 causes the thread manager 133 to generate a thread, allocates the resource to the thread, and reflects the allocation result in the resource usage of each resource constraint management table (910, 930, 950, and 970). .

In S92, the system module 120 starts executing the thread and ends the resource constraint management process.

In S93, the system module 120 saves the thread generation information to be referenced for generating a thread in the thread generation suspension management table 990.

In S94, the system module 120 sets the own thread as a suspended state where the generation of the thread is suspended, and ends the process.

The thread generation suspension management table 990 includes a task ID 991, a parent thread ID 992, a child thread ID 993, a time 994, and thread generation information 995 as information on threads that are suspended from thread generation. Various information is as follows.
(*) Task ID 991 indicates the ID of the task.
(*) The parent thread ID 992 indicates the ID of a parent thread (parent thread) that generates another thread.
(*) Child thread ID 993 indicates the ID of a thread (child thread) that is a child generated from the thread.
(*) Time 994 indicates the time when the generation of the thread is suspended.
(*) The thread generation information 995 is information necessary for generating a thread (for example, information including a reference indicating a record referred to by a child thread).

The process in the upper right of FIG. 8H (steps S95 to S96) is a process for stopping the thread when the process executed by the thread is completed.

In S95, the system module 120 stops the executing thread.

In S96, the system module 120 releases resources allocated to its own thread. That is, the system module 120 deletes the amount of resources allocated to its own thread from the resource amounts (number of threads, main memory allocation amount) managed by resource usage such as the resource constraint management table 910. As a result, the resources allocated to the own thread can be allocated to other threads.

The processing at the lower right of FIG. 8H (steps S97 to S99) is processing that is executed when it is determined, for example, that there are sufficient resources to generate a thread based on the resource constraint management table 910. This procedure may be driven, for example, by means such as an interrupt or a signal when resources are released in S96.

In S97, the system module 120 selects the thread generation information managed in the thread generation suspension management table 990. The thread generation information to be selected may be, for example, the oldest thread generation information reserved.

In S98, the system module 120 resumes the parent thread that generates a thread based on the selected thread generation information.

In S99, the system module 120 executes child thread generation again based on the thread generation information. With this process, when there are sufficient resources to generate a thread, a thread that has been suspended can be generated.

FIG. 9A shows a first example of tasks according to the first embodiment.

FIG. 9A shows map task # 1111 of map process # 111 of map reduce job # 1 (corresponding to the job execution plan of FIG. 1A). Here, it is assumed that the record of # 1001 of the input data set # 1 includes references to 10 records # 2001 to # 2010 of the input data set # 2. Also, map task # 1111 acquires the record of # 1001 of input data set # 1, and acquires the record of input data set # 2.

When the calculation node 100 executes the map task # 1111, the record of # 1001 of the input data set # 1 is referred to. If the record satisfies a predetermined condition, the reference included in the record is used. Thus, records # 2001 to # 2010 of the input data set # 2 are acquired.

FIG. 9B shows an example of thread generation in the task shown in the first example. FIG. 9B illustrates thread generation when the task # 1111 illustrated in FIG. 9A is not performed when the resource constraint management process illustrated in FIG. 8H is not performed. Note that the notation in FIG. 9B follows the following rules.
(*) The horizontal axis represents time.
(*) A round rectangle with a long side in the figure means a series of processing by one thread. The left end of the rounded rectangle represents the time at which the processing by the thread is started, and the right end of the rounded rectangle represents the time at which the processing by the thread ends.
The value inside the (*) rounded rectangle represents information (for example, record ID) indicating a record that is read along with the processing corresponding to the thread.
(*) The number of threads that can be executed simultaneously for the thread that acquires the record is “8”.

When the resource constraint management process shown in FIG. 8H is not performed, when the thread 10a that executes the process by acquiring the record of # 1001 of the input data set # 1 is executed, the record of # 1001 is acquired by the thread 10a. Then, based on the reference included in the record of # 1001, the thread 10b that acquires and processes the record of # 2001 of the input data set # 2 is generated and executed, and # of the input data set # 2 A thread 10c that acquires and records a record 2002 is generated and executed. Similarly, a thread 10d, a thread 10e, a thread 10f, a thread 10g, and a thread 10h are generated and executed. At this time, the same number of threads as “8”, which is the number of threads that can be executed simultaneously, are executed.

Thereafter, the thread 10a further acquires and processes the # 2008 record of the input data set # 2 and performs processing, the thread 10j that acquires and processes the record of # 2009 of the input data set # 2, and A thread 10k that acquires and processes a record of # 2010 of the input data set # 2 is generated and executed. The main memory is allocated beyond the available amount of main memory, and thrashing occurs, resulting in a long execution time for these threads.

FIG. 9C illustrates an example of thread generation in the task illustrated in the first example according to the first embodiment. FIG. 9C shows thread generation when the resource constraint management process shown in FIG. 8H is performed when the task # 1111 shown in FIG. 9A is executed. The notation in FIG. 9C is the same as the rule in FIG. 9B.

When the thread 10a that executes processing by acquiring the record # 1001 of the input data set # 1 is executed in the computation node 100, the thread # 10a acquires the record # 1001 and includes it in the record # 1001. The thread 10b that acquires and processes the record of # 2001 of the input data set # 2 is generated and executed based on the received reference, and acquires and processes the record of # 2002 of the input data set # 2. A thread 10c is generated and executed. Similarly, a thread 10d, a thread 10e, a thread 10f, a thread 10g, and a thread 10h are generated and executed. At this time, the same number of threads as “8”, which is the number of threads that can be executed simultaneously, are executed.

Thereafter, since it is determined in step S90 of the resource constraint management process that there are not enough resources, the thread 10a is put on hold without generating a new thread. When the execution of the thread 10b is completed, one new thread can be executed. Therefore, the execution of the thread 10a is resumed in step S98, and the record of # 2008 of the input data set # 2 is detected in step S99. A thread 10i is generated that performs processing by acquiring. Similarly, when the execution of the thread 10c is finished, the execution of the thread 10a is resumed, and the thread 10j that acquires and processes the record of # 2009 of the input data set # 2 is generated and executed. When the execution of the thread 10d is completed, the execution of the thread 10a is resumed, and the thread 10k that acquires and processes the record of # 2010 of the input data set # 2 is generated and executed.

As a result, when executing a plurality of threads, it becomes possible to allocate main memory within the range of the amount of main memory that can be used, so that thrashing can be prevented, and the execution time of the entire task is shown in FIG. 9B. Compared to the case, it can be shortened.

Next, the generation of threads when multiple tasks are executed in parallel will be described.

FIG. 9D shows a second example of tasks according to the second embodiment.

FIG. 9D shows the map task # 1111 of the map process # 111 of the map reduce job # 1 and the map task # 2111 of the map process # 211 of the map reduce job # 2. It is assumed that map task # 1111 and map task # 2111 are executed in parallel.

Suppose that the record of # 1001 of the input data set # 1 includes references to 10 records from # 2001 to # 2010 of the input data set # 2. The map task # 1111 acquires the record of # 1001 of the input data set # 1, and acquires the data of the input data set # 2 corresponding to this record.

When executing the map task # 1111, the system module 120 refers to the record of # 1001 of the input data set # 1, and if the record satisfies a predetermined condition, the system module 120 uses the reference included in the record. Thus, records # 2001 to # 2010 of the input data set # 2 are acquired.

Also, it is assumed that the record of # 5001 of the input data set # 5 includes references to 10 records from # 6001 to # 6010 of the input data set # 6. Also, map task # 2111 acquires a record of # 5001 of input data set # 5, and acquires a record of input data set # 6 corresponding to this record.

When executing the map task # 2111, the system module 120 refers to the record of # 5001 of the input data set # 5. If the record satisfies a predetermined condition, the system module 120 uses the reference included in the record. Thus, records # 6001 to # 6010 of the input data set # 6 are acquired.

FIG. 9E illustrates an example of thread generation in the task illustrated in the second example according to the first embodiment. FIG. 9E shows thread generation when the resource constraint management process shown in FIG. 8H is performed when task # 1111 and task # 2111 shown in FIG. 9D are executed in parallel. Here, it is assumed that the main memory area available for task # 1111 is for five threads, and the main memory area available for task # 2111 is for three threads. Note that the notation in FIG. 9E is the same as the rule in FIG. 9B.

When the thread 11a that executes processing by acquiring the record of # 1001 of the input data set # 1 is executed in the calculation node 100, the thread 11a acquires the record of # 1001 and is included in the record of # 1001. The thread 11b that acquires and processes the record of # 2001 of the input data set # 2 is generated and executed based on the received reference, and acquires and processes the record of # 2002 of the input data set # 2. A thread 11c is generated and executed. Similarly, a thread 10d and a thread 10e are generated and executed. At this time, an area corresponding to 5 threads in the main memory that can be used by the task # 1111 is used.

Thereafter, since it is determined in step S90 of the resource constraint management process that there are not enough resources, the thread 11a does not generate a new thread, and its own thread is put on hold. When the execution of the thread 11b is completed, one new thread can be executed. Therefore, the execution of the thread 11a is resumed in step S98, and the record of # 2005 of the input data set # 2 in step S99. A thread 11f is generated that performs processing by acquiring. Similarly, when the execution of the thread 11c is finished, the execution of the thread 11a is resumed, and the thread 11g for acquiring and processing the record of # 2006 of the input data set # 2 is generated and executed. When the execution of the thread 11d is finished, the execution of the thread 11a is resumed, and the thread 11h that acquires and processes the record of # 2007 of the input data set # 2 is generated, and the execution of the thread 11d is finished. In this case, the execution of the thread 11a is resumed, and the thread 11i for obtaining and processing the record of # 2008 of the input data set # 2 is generated. When the execution of the thread 11f is completed, the thread 11a Execution is resumed, and a thread 11j is generated that acquires and processes the record of # 2009 of the input data set # 2. , If the execution of the thread 11g is completed, the execution of the thread 11a is resumed, the input data set # 2 # thread 11k to perform acquisition and processing records 2010 are generated.

On the other hand, when the thread 12a that performs processing by acquiring the record of # 5001 of the input data set # 5 is executed in parallel, the record of # 5001 is acquired by the thread 12a and included in the record of # 5001. The thread 12b that acquires and processes the record of # 6001 of the input data set # 6 is generated and executed based on the received reference, and acquires and processes the record of # 6002 of the input data set # 6 A thread 12c is generated and executed. At this time, an area corresponding to three threads of the main memory that can be used for task # 2111 is used.

Thereafter, since it is determined in step S90 of the resource constraint management process that there are not enough resources, the thread 12a does not generate a new thread, and its own thread is put on hold. When the execution of the thread 12b ends, one new thread can be executed. Therefore, the execution of the thread 12a is resumed in step S98, and the record of # 6003 of the input data set # 6 is detected in step S99. A thread 12d is generated that performs processing by acquiring. Similarly, when the execution of the thread 12c is finished, a thread 12e for obtaining and processing the record of # 6004 of the input data set # 6 is generated, and when the execution of the thread 12d is finished, the input When a thread 12f for obtaining and processing the record of # 6005 of the data set # 6 is generated and the execution of the thread 12e is finished, the record of # 6006 of the input data set # 6 is obtained and processed. When the thread 12g is generated and the execution of the thread 12f is finished, a thread 12h for obtaining and processing the record of # 6007 of the input data set # 6 is generated, and when the execution of the thread 12g is finished. , A thread 12i for obtaining and processing the record of # 6008 of the input data set # 6 is generated, and the thread 12h When the execution is finished, a thread 12j for obtaining and processing the record of # 6009 of the input data set # 6 is generated, and when the execution of the thread 12i is finished, # 6010 of the input data set # 6. A thread 12k that acquires and processes the record is generated. In this way, a plurality of tasks can be executed in parallel.

As a result, when multiple threads are executed in multiple tasks, it becomes possible to allocate main memory within the range of the amount of main memory available for each task, so that thrashing can be prevented. The execution time can be prevented from being prolonged due to thrashing.

In the above, examples of the number of threads and main memory are shown as resource constraints, but the present invention is not limited to this example. For example, with respect to processor execution time, storage input / output throughput, network transmission throughput, and the like, the same effect can be expected by similarly restricting resources.

As mentioned above, although several examples were described, it cannot be overemphasized that this invention can be variously changed in the range which is not limited to these Examples and does not deviate from the summary.

Various implementation forms are conceivable as threads that are dynamically generated when executing parallel data processing. For example, a process or a kernel level thread (such as a native POSIX thread or a lightweight process) Threads managed by the kernel), user-level threads (threads managed by user programs and libraries such as fiber), and so that they can be executed in parallel with others It may be a collection of fixed procedures to be managed (for example, a function pointer managed by an appropriate structure), or a combination of these.

In this specification, the unit of data handled by parallel data processing is a record, but the record may be arbitrary data. For example, it may be a collection of a certain number of columns, a collection of a variable number of columns, a simple text, a byte string, an image, sound, etc. The multimedia content may also be a collection of these.

100 ... Compute node 110 ... Application 120 ... System module

Claims

A parallel data processing system included in one computer in a computer system that executes data processing in parallel on a plurality of computers,
A parallel data processing execution unit that reads data from a data group that includes a first data group that includes a plurality of first data and a second data group that includes a plurality of second data, and executes processing;
The parallel data processing execution unit
(A) Reading the first data from the first data group, and acquiring a first value from the first data based on the first format information acquired from the application,
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data of the first data group;
(D) A parallel data processing system that executes a plurality of the threads in parallel.
The parallel data processing execution unit
(E) The second data is read by executing a thread in (D), and a second value is acquired from the second data based on second format information acquired from an application. Parallel data processing system.
The parallel data processing execution unit
The parallel data processing system according to claim 2, wherein the second condition acquired from the application is evaluated based on the second value.
The parallel data processing execution unit
The parallel data processing system according to claim 2, wherein output data is generated from the second value acquired from one or more of the second data.
The parallel data processing execution unit
The parallel data processing system according to claim 1, wherein a first condition acquired from the application is evaluated based on the first value, and the process (B) is executed when the first condition is satisfied.
The parallel data processing system according to claim 1, wherein the first reference information includes information for specifying a physical position where the second data is stored in the second data group.
The parallel data processing system according to claim 1, wherein the first reference information includes information for searching for the second data in the second data group.
The second data of at least a part of the second data group is stored in a storage device of another computer connected via a network,
The parallel data processing execution unit
When the second data is acquired from the another computer connected via the network by executing the thread, an acquisition request is transmitted to the another computer, and the storage device transmits the second data. The parallel data processing system according to claim 1, wherein the second data is acquired.
The parallel data processing execution unit
9. The plurality of second data is acquired by sending a block acquisition request, which is a collection of a plurality of acquisition requests for the same computer generated by execution of a plurality of threads, to the other computer. Parallel data processing system described in 1.
The first format information is a program code,
The parallel data processing execution unit
Receiving catalog information necessary for creating first format information described in a predetermined markup language from a user;
The parallel data processing system according to claim 1, wherein the first format information is created based on the catalog information.
When the parallel data processing execution unit generates a new thread based on resource constraint information related to thread generation in a computer having the parallel data processing execution unit, the parallel data processing execution unit is used for execution of the thread in the calculation node constituting the parallel data processing execution unit. The parallel data processing system according to claim 1, wherein the generation of the thread is suspended when it is determined that a resource amount to be exceeded exceeds a constraint.
When the parallel data processing execution unit generates a new thread based on resource constraint information regarding generation of a thread in a process responsible for a part of the parallel data processing, a resource used for executing the thread in the process The parallel data processing system according to claim 1, wherein the generation of the thread is suspended when the amount exceeds a constraint.
It further has a reception unit that receives processing instructions from the application,
The instructions from the application define the procedure,
The parallel data processing execution unit, upon receiving the instruction, executes the steps (A) to (D), so that even if the instruction defines the procedure, an out-of-order process that does not depend on the procedure Run the
The parallel data processing system according to claim 1.
A computer in a computer system that executes data processing in parallel on a plurality of computers,
A communication interface device for communicating with another computer in the computer system;
A control device connected to the communication interface device, which reads data from a data group including a first data group including a plurality of first data and a second data group including a plurality of second data, and executes processing; Have
The control device is
(A) Reading the first data from the first data group, and acquiring a first value from the first data based on the first format information acquired from the application,
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data of the first data group;
(D) A computer that executes a plurality of the threads in parallel.
A parallel data processing method in a computer system that executes data processing in parallel on a plurality of computers,
(A) Reading the first data from the first data group of the data group including a first data group including a plurality of first data and a second data group including a plurality of second data, and from an application Obtaining a first value from the first data based on the obtained first format information;
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data of the first data group;
(D) A parallel data processing method for executing a plurality of threads in parallel.
Have multiple calculators,
Each calculator
A parallel data processing system that reads data from a data group that includes a first data group that includes a plurality of first data and a second data group that includes a plurality of second data, and executes processing;
The parallel data processing system of each computer is
(A) Reading the first data from the first data group, and acquiring a first value from the first data based on the first format information acquired from the application,
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data of the first data group;
(D) A computer system that executes a plurality of the threads in parallel.
A computer program executed by a computer in a computer system that executes data processing in parallel on a plurality of computers,
(A) Reading the first data from the first data group of the data group including a first data group including a plurality of first data and a second data group including a plurality of second data, and from an application Obtaining a first value from the first data based on the obtained first format information;
(B) generating one or more threads for reading each of the one or more second data corresponding to the first value from the second data group based on the first reference information acquired from the application;
(C) performing (A) to (B) on one or more first data of the first data group;
(D) A computer program that causes the computer to execute a plurality of threads in parallel.