JP6940325B2 - Distributed processing system, distributed processing method, and distributed processing program - Google Patents

Description

The present invention relates to a distributed processing system, a distributed processing method, and a distributed processing program.

In recent years, big data technology, which obtains new knowledge by analyzing a large amount of data and utilizes it, has been attracting attention. As a method for analyzing such a large amount of data, there are various methods such as statistical analysis and machine learning. In addition, other techniques such as data shaping are being combined. When combining these multiple methods, for example, the data processing order is determined in advance in association with the analysis flow, and as soon as the prerequisite processing is completed, the data obtained as a result of the processing is displayed as follows. The final result can be obtained by using it as the input data of the processing of. If appropriate findings cannot be obtained by referring to the results, a new analysis flow is determined by changing the method and conditions. As described above, analysis of a large amount of data requires repeated trial and error, so that the amount of data processed is large and the time required for analysis is often long.

Therefore, in general, in order to shorten the processing time, instead of processing the data with a single computer, a plurality of computers are distributed to perform parallel processing. For example, the data to be analyzed is stored in a system called a data lake using a distributed file system or the like. Then, a plurality of computers divide the data in this data lake into a plurality of sections and read it, format or analyze the read data, store the analyzed data in the data lake or the like, and display the contents to the user. .. Furthermore, since the data analysis process is a temporary process, a virtual machine system that is easy to build will be used. That is, by starting each virtual machine when data analysis processing is required and causing them to perform the analysis processing in a distributed manner, it is possible to reduce the amount of resources used for the analysis processing.

When allocating resources to virtual machines, there is known a method of managing allocation so that resource allocation is not performed to each virtual machine in excess of a predetermined amount so that the operation does not become unstable. For example, in Patent Document 1, the computer management system satisfies the policy value when the average resource usage rate of some virtual machines exceeds a predetermined policy value. Allocate resources from other virtual machines that meet the policy value, and if the allocated resource capacity is larger than the surplus resources of the physical server that has the virtual machine that takes over the resources, allocate the resources so that the resource capacity of the physical server is not exceeded. It is stated that a change instruction will be issued.

Japanese Unexamined Patent Publication No. 2014-130413

However, when a plurality of data analyzes are executed on a virtual machine, the allocation of resources used for processing differs depending on the data analysis content, and as a result, data processing may not be performed efficiently. In the technique described in Patent Document 1, since the execution order constraint of each data analysis and the resource capacity constraint of the physical server are fixedly held during the analysis process, the allocation of resources between each data analysis is also possible. It becomes fixed. As a result, data processing becomes unstable, which may cause problems such as processing delays.

The present invention has been made in view of such a current situation, and an object of the present invention is to provide a distributed processing system, a distributed processing method, and a distributed processing program capable of stably distributing and processing data.

One of the present inventions for solving the above problems is a predetermined process including a plurality of virtual machines and including a plurality of process flows for processing predetermined data composed of a plurality of partitions for each partition. the processing flow of the processing flow can be executed in parallel by a assigned to at least one of the plurality of virtual machines, a distributed processing system having a processor and a memory, said plurality of processing on data of the partition A flow table storage unit that stores the processing order of flows , a processing load calculation unit that calculates the load on the virtual machine due to the processing of data in the partition by the processing flow for each partition, and the current processing load of each processing flow. execution state, the processing flow each of said processing flow of a processing order, and based on the load calculated for each said processing flow are executed in parallel, and flow to determine the combination of the virtual machine running the process flow A management unit and a machine control unit that causes each virtual machine to execute parallel processing indicated by the determined combination are provided, and the processing load calculation unit uses the load as an estimated time for executing the processing flow. Is calculated for each of the processing flows, and when there are a plurality of the processing flows executed in parallel, the flow management unit allocates the virtual to each of the plurality of processing flows based on the calculated predicted time. If there is a processing flow for which the estimated time has not been calculated among the plurality of processing flows executed in parallel by determining the priority regarding the machine or its allocation, each of the plurality of processing flows is executed. Make the number of virtual machines equal to each other .

According to the present invention, data can be stably distributed and processed.

FIG. 1 is a diagram showing an example of the configuration of the distributed processing system 100 according to the present embodiment. FIG. 2 is a diagram showing an example of a hardware configuration included in each information processing device (virtual machine management server 101, virtual machine execution server 102, data lake 104, and user operation terminal 105) in the distributed processing system 100. FIG. 3 is a diagram illustrating an example of a function provided in each information processing device. FIG. 4 is a diagram showing an example of the analysis flow table 300. FIG. 5 is a diagram showing an example of the processing state management table 400. FIG. 6 is a diagram showing an example of the VM management table 500. FIG. 7 is a sequence diagram showing an example of analysis processing performed by the distributed processing system 100. FIG. 8 is a flowchart illustrating the details of the flow execution target determination process. FIG. 9 is a flowchart showing details of the virtual machine allocation determination process. FIG. 10 is an example of a screen displayed by the user operation terminal 105 showing the progress or result of the analysis process. FIG. 11 is a diagram showing an example of time-series changes in the usage status of the hardware resources of the virtual machine execution server 102 and the data lake 104 when the analysis process is executed in the conventional distributed processing system. FIG. 12 is a diagram showing an example of time-series changes in the hardware resource usage status of the virtual machine execution server 102 and the data lake 104 when the analysis process is executed in the distributed processing system 100 of the present embodiment.

<System configuration>
FIG. 1 is a diagram showing an example of the configuration of the distributed processing system 100 according to the present embodiment. The distributed processing system 100 is an information processing system that performs processing (hereinafter referred to as analysis processing) for analyzing changes in predetermined data (hereinafter referred to as target data).

The target data is, for example, data on changes in temperature, pressure, and velocity over time, and is so-called big data that exists in large quantities. This target data is composed of a plurality of data partitions, for example, data delimited by a predetermined time zone.

As shown in FIG. 1, the distributed processing system 100 includes a data lake 104, which is a database for storing target data, and at least one virtual machine 103 (so-called virtual machine 103) that reads target data from the data lake 104 and executes analysis processing. A virtual machine execution server 102 including a server), a virtual machine management server 101 that gives instructions regarding analysis processing to each of the virtual machines 103, and an administrator, a user, or the like (hereinafter referred to as a user) of the distributed processing system 100. It is configured to include a user operation terminal 105 for displaying instructions related to analysis processing and information related to analysis processing to be used.

Between the information processing devices of the virtual machine management server 101, the virtual machine execution server 102, the data lake 104, and the user operation terminal 105, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, etc. It is communicably connected by a network 108 composed of a dedicated line or the like.

The analysis process includes a plurality of processing units (hereinafter, also referred to as analysis flows) for processing the target data. The analysis flow may include various processes for processing and analyzing big data, such as various statistical analysis processes and machine learning-related processes.

At least two or more virtual machines 103 are provided in the distributed processing system 100, and the distributed processing system 100 performs a plurality of virtual machines for each analysis flow (more specifically, data processing in each data partition) in the analysis processing. By assigning to at least one of the machines 103, each analysis flow can be executed in parallel by the plurality of virtual machines 103.

Note that FIG. 2 is a diagram showing an example of the hardware configuration included in each information processing device (virtual machine management server 101, virtual machine execution server 102, data lake 104, and user operation terminal 105) in the distributed processing system 100. .. As shown in the figure, each information processing device includes a processor 51 such as a CPU (Central Processing Unit), a main storage device 52 such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and an HDD (Hard Disk).
Auxiliary storage device 53 such as Drive) and SSD (Solid State Drive), an input device 54 including a keyboard, a mouse, a touch panel and the like, and an output device 55 including a monitor (display) and the like are provided.

Although not shown in FIG. 1, an information processing device (various servers, sensors, etc.) that measures the time change of the target data is connected to the data lake 104, and these information processing devices use the measured value (the measured value (). The target data) may be transmitted to the data lake 104 and stored.

Next, the functions provided by each information processing device will be described.
<Function>
FIG. 3 is a diagram illustrating an example of a function provided in each information processing device.
First, the virtual machine management server 101 includes a flow table storage unit 211, a processing state management table storage unit 212, a processing load calculation unit 201, a flow management unit 202, a machine management table storage unit 213, and a machine control unit 203.

The flow table storage unit 211 stores the processing order of the plurality of processing units for the data in the partition.

That is, the flow table storage unit 211 includes an analysis flow table 300. The details of the analysis flow table 300 will be described later.

The processing state management table storage unit 212 stores the current execution state of each analysis flow in the processing state management table 400. Details of the processing state management table 400 will be described later.

The machine management table storage unit 213 stores the constraint conditions related to the parallel execution of the processing unit by the information processing device (virtual machine 103).

Specifically, the machine management table storage unit 213 stores the maximum number of information processing devices capable of executing the processing unit in parallel as the constraint condition.

That is, the machine management table storage unit 213 includes a VM management table 500. Details of the VM management table 500 will be described later.

The processing load calculation unit 201 calculates the load on the information processing device (virtual machine 103) due to the processing of the data in the partition performed by the processing unit (analysis flow) for each partition.

Specifically, the processing load calculation unit 201 calculates, as the load, the estimated time for executing the processing of the processing unit for each of the processing units.

The flow management unit 202 parallelly based on the current execution state of each of the processing units (analysis flow), the processing order of each of the processing units, and the load calculated by the processing load calculation unit 201 for each of the processing units. The combination of the processing unit to be executed and the information processing device that executes the processing unit is determined.

Specifically, the flow management unit 202 determines the information processing device satisfying the constraint condition as the information processing device (virtual machine 103) that executes the processing units in parallel.

Further, when there are a plurality of the processing units (analysis flows) executed in parallel, the flow management unit 202 refers to each of the plurality of processing units based on the predicted time calculated by the processing load calculation unit 201. The information processing device (virtual machine 103) to be assigned or the priority regarding the allocation is determined.

Further, the flow management unit 202 executes each of the plurality of processing units executed in parallel when there is the processing unit for which the predicted time has not been calculated. Make the number of information processing devices equal to each other.

Further, when the flow management unit 202 determines the processing unit to be executed in parallel, if there are a plurality of data partitions that can be processed by the processing unit, a predetermined first processing is performed. It is determined that the processing unit processes only the section of the data to be processed.

The machine control unit 203 causes each information processing device to execute the parallel processing indicated by the combination determined by the flow management unit 202.

Next, the virtual machine execution server 102 operates the virtual machine 103. The virtual machine 103 includes a transmission / reception unit 204 and an analysis processing unit 205.

The transmission / reception unit 204 transmits / receives data related to the analysis process. The analysis processing unit 205 has various statistical analysis functions, and executes each analysis flow in the analysis processing by, for example, extracting, analyzing, and storing data.

The data lake 104 includes a data storage unit 206 and a data read / write unit 207. The data storage unit 206 stores the target data, and also transmits / receives various data including the target data in response to a data read request or write request from the virtual machine 103. The data reading / writing unit 207 reads various data including the target data and writes the data.

The target data stored in the data lake 104 is, for example, a set of a plurality of data according to a predetermined format. For example, when the target data is time-series data, the target data is divided into the plurality of data partitions according to the time or time zone. In the analysis flow, a predetermined analysis process is performed on the data in each data section.

In the following, in the analysis flow, the process of processing the data of a certain data section is referred to as a section flow.

The user operation terminal 105 includes an output unit 208. The output unit 208 outputs information regarding the result of the processing of the processing unit executed by each of the information processing devices or the amount of data input / output generated by the processing of the processing unit.

Next, each table (database) stored in the distributed processing system 100 will be described.

<Analysis flow table>
FIG. 4 is a diagram showing an example of the analysis flow table 300. As shown in the figure, in the analysis flow table 300, the flow name 611 in which the analysis flow identification information (hereinafter referred to as the flow name) is stored, and the data (or its type) input to the analysis flow indicated by the flow name 611. Input 612 that stores information that identifies It is composed of at least one or more records having each item of the output 614 in which the information specifying the data (or the type thereof) output from the analysis flow indicated by the flow name 611 is stored.

In the following, the data stored in the input 612 when the analysis flow indicated by the flow name 611 is the first process to be executed in the analysis process is referred to as initial input data. Further, when the analysis flow indicated by the flow name 611 is the last process to be executed in the analysis process, the data stored in the output 614 is referred to as the final output data. The data stored in the input 612 or the output 614 in other cases is referred to as intermediate data.

In the example of the figure, "analysis process A" reads (receives) the target data stored in the data lake 104 from the beginning, and "analysis process B" is "intermediate data A" output by "analysis process A". Is read (received), and "analysis process C" reads (receives) "intermediate data B" output by "analysis process B". The data input to "analysis process A" is the initial input data, the data output by "analysis process A" and "analysis process B" is intermediate data, and the data output by "analysis process C" is This is the final output data.

As described above, the analysis flow table 300 defines the processing order between the analysis flows in the analysis process by storing the information that identifies the data transmitted or received by each analysis flow. The contents of the analysis flow table 300 are input in advance by the user, for example, before the analysis process.

<Processing status management table>
FIG. 5 is a diagram showing an example of the processing state management table 400. The processing state management table 400 stores the execution state of the section flow in the analysis flow on the premise of the processing order of the analysis flow defined by the analysis flow table 300.

That is, in the processing state management table 400, the flow name 621 in which the flow name is stored, the section name 622 in which the identification information of the section flow in the analysis flow indicated by the flow name 621 (hereinafter referred to as the section name) is stored, and the section name 622 are stored. The current execution state of the interval flow indicated by (for example, whether it is being executed (“execution”), whether the execution is completed (“execution completed”), or whether it is not being executed but is in a state where it can be executed (“execution completed”). Execution state 623, which stores information indicating whether execution is impossible ("executable"), or because the data required to start execution has not been generated ("executable"), etc. When "execution completed" is stored in the state 623, the processing time 624 in which the time required for the execution (hereinafter referred to as the execution time) is stored, and the time required to execute the section flow indicated by the section name 622. It is composed of at least one or more records having each item of the estimated time 625 in which the estimated time (estimated time) of is stored. In the predicted time 625, the predicted time is stored when the execution time is not stored in the processing time 624 (that is, when the execution of the section flow indicated by the section name 622 is not completed).

The predicted time is calculated based on, for example, the execution time of another section flow executed in the past, as will be described later.

In the example of the figure, since the execution of "section A", which is the section flow of "analysis process A", has been completed, "12 minutes" is recorded in the processing time 624. Further, in the "section C" of the "analysis process A", since the execution of the "section A" in which the same type of statistical analysis has been performed has been completed, the same "12 minutes" as described above is recorded in the predicted time 625. Has been done. The predicted time 625 stores the predicted time when a single virtual machine 103 executes the section flow for accurate comparison of the predicted time between the section flows.

The contents of the processing state management table 400 are updated at a predetermined timing or at a predetermined time interval.

<VM management table>
FIG. 6 is a diagram showing an example of the VM management table 500. As shown in the figure, the VM management table 500 is used in the virtual machine execution server 102 indicated by the execution server name 711 and the execution server name 711 in which the identification information (hereinafter referred to as the execution server name) of the virtual machine execution server 102 is stored. Maximum number of VMs that can be assigned 712, which stores information indicating the maximum number of virtual machines (hereinafter referred to as the maximum number) that can be assigned the analysis flow (section flow in) to the virtual machine 103, and execution. At least one or more having each item of the allocated VM number 713 in which information indicating the number of virtual machines 103 currently allocated to the virtual machine execution server 102 indicated by the server name 711 (hereinafter referred to as the current number) is stored. Consists of records.

The content of the allocated VM number 713 of the VM management table 500 is updated at a predetermined timing (for example, at a predetermined time interval, when the virtual machine 103 is started, or when the virtual machine 103 is stopped). Other items are entered by the user, for example, before the analysis process is executed.

In the example of the figure, three virtual machine execution servers 102 of "execution server A", "execution server B", and "execution server C" are registered, and the maximum number of all virtual machine execution servers 102 is set. It is "4".

The function of each information processing device described above is that the hardware of each information processing device or the processor 51 of each information processing device reads out each program stored in the main storage device 52 and the auxiliary storage device 53. It is realized by executing. This program is, for example, a secondary storage device, a non-volatile semiconductor memory, a hard disk drive, a storage device such as an SSD, or a non-temporary data storage medium such as an IC card, an SD card, or a DVD that can be read by a computer. Stored in.

Next, each process performed by the distributed processing system 100 will be described.
<Analysis processing>
FIG. 7 is a sequence diagram showing an example of analysis processing performed by the distributed processing system 100. This process is started, for example, when information for designating the analysis process to be executed is input from the user operation terminal 105 to the virtual machine management server 101.

First, the flow management unit 202 of the virtual machine management server 101 executes an analysis flow (hereinafter referred to as a target flow) and its section flow (hereinafter referred to as a target section flow) executed in parallel by the virtual machine 103 in the designated analysis process. ) Is executed (hereinafter, referred to as a flow execution target determination process) (S301). The details of this process will be described later.

Next, the flow management unit 202 transmits to the machine control unit 203 an instruction for assigning processing to each virtual machine 103, which is accompanied by information on each target flow and target section flow specified in S301 (S302). ..

Upon receiving the above instruction, the machine control unit 203 determines to which of the virtual machines 103 each target section flow in each target flow is assigned to execute parallel processing (hereinafter, virtual). The machine allocation determination process) is executed (S303). The details of this process will be described later. After the processing is completed, the machine control unit 203 updates the VM management table 500.

The machine control unit 203 transmits an instruction to start the virtual machine 103 to each virtual machine 103 to which processing is assigned by S303 (S304). Then, the flow management unit 202 transmits to each virtual machine 103 an instruction to execute the target section flow (execution of parallel processing) assigned to the virtual machine 103 to each virtual machine 103 started by S304. (S305).

The transmission / reception unit 204 of each virtual machine 103 that has received the transmission of the instruction from the virtual machine management server 101 instructs the analysis processing unit 205 to execute the target section flow (S306).

Each analysis processing unit 205 of each virtual machine 103 instructed by S306 executes the target section flow assigned to itself (S307).

For example, a certain analysis processing unit 205 reads the initial input data of the target flow from the data storage unit 206 of the data lake 104 and executes the target section flow. Further, the other analysis processing unit 205 executes the target section flow based on the intermediate data. Further, the analysis processing unit 205 executes the target section flow, outputs the final output data, and transmits this to the data storage unit 206 of the data lake 104.

When a plurality of virtual machines 103 (analysis processing unit 205) execute the same target section flow of the same target flow in parallel, each analysis processing unit 205 divides, for example, initial input data or intermediate data. Then, it is assigned to the analysis processing unit 205, and each analysis processing unit 205 executes the target section flow in parallel based on the assigned data.

When each analysis processing unit 205 of each virtual machine 103 finishes executing the target section flow assigned to itself, it transmits a notification to that effect to the transmission / reception unit 204 (S308).

When the transmission / reception unit 204 of each virtual machine 103 receives the above notification from the analysis processing unit 205, the transmission / reception unit 204 transmits a notification to the effect that the execution of the target section flow is completed to the flow management unit 202 of the virtual machine management server 101 (S309). ).

When the flow management unit 202 receives the above-mentioned end notification from all the virtual machines 103 that have sent the execution instruction in S305, the flow management unit 202 transmits an instruction to release the allocation instructed in S302 to the machine control unit 203 (. S310). Upon receiving this instruction, the machine control unit 203 stops the operation (execution) of all the virtual machines 103 that have been assigned (S311).

After that, the machine control unit 203 acquires the execution state of each analysis flow (section flow) of each of the current virtual machines 103 from each virtual machine 103, and updates the VM management table 500 based on the acquired state (S312). Specifically, for example, the machine control unit 203 updates the value of the allocated VM number 713 of the VM management table 500 with the number of virtual machines 103 currently executing the analysis flow (interval flow).

Next, the flow management unit 202 generates and updates information regarding the execution state of the analysis process (S313).

Specifically, first, the flow management unit 202 calculates the execution time of the target section flow performed by each virtual machine 103. For example, the flow management unit 202 calculates the time from the time when the processing of S304 is performed to the time when the processing of S309 is performed as the execution time.

Further, the flow management unit 202 instructs the processing load calculation unit 201 to calculate the estimated time of the section flow that has not yet been executed based on the calculated execution time of the target section flow. Specifically, for example, the processing load calculation unit 201 performs the same type of statistical analysis as the target section flow for which the execution time is calculated, and determines the estimated time of each section flow of the unexecuted analysis flow as the execution time. To be the same as. When the target section flow for which the execution time is calculated is executed by a plurality of virtual machines 103, the estimated time based on this is converted into the time when the single virtual machine 103 executes the process.

The flow management unit 202 stores the execution time and the predicted time calculated as described above in the processing state management table 400. Specifically, the processing status management table storage unit 212 stores the calculated execution time of each target section flow in the processing time 624 of each record of the processing status management table 400, and stores each estimated time in the processing status management table 400. It is stored in the estimated time 625 of each record.

The above processes from S301 to S313 are repeated until all the analysis flows complete the execution. This completes the analysis process.

Next, the details of the flow execution target determination process and the virtual machine allocation determination process will be described.

<Flow execution target judgment processing>
FIG. 8 is a flowchart illustrating the details of the flow execution target determination process. As shown in the figure, the flow management unit 202 first determines whether or not there is only one analysis flow that can be executed at present (S401). Specifically, for example, the flow management unit 202 confirms the number of records in which "executable" is stored in the execution state 623 of each record in the processing state management table 400. As a result, the number of flows that can be executed at the same time can be determined without breaking the restrictions on the execution order of each analysis flow.

When there is only one analysis flow that can be executed at present (S401: YES), the flow management unit 202 stores the analysis flow as a target flow (S402). For example, the flow management unit 202 stores a record in the processing state management table 400 in which the target flow is stored. After that, the processing of S410 is performed.

On the other hand, when there are a plurality of analysis flows that can be executed at present (S401: NO), the flow management unit 202 determines whether or not the predicted times of all the plurality of analysis flows have been calculated (S403). Specifically, for example, the flow management unit 202 refers to the predicted time 625 of each record of the plurality of analysis flows in the processing state management table 400, and confirms whether or not the predicted time is stored.

When the predicted times of all the plurality of analysis flows have been calculated (S403: YES), the flow management unit 202 indicates that the number or priority of the virtual machines 103 that execute the analysis flow is appropriately weighted. , All analysis flows enabled in S401 (all analysis flows for which the predicted time is calculated in S403) are stored as target flows (S404), and then stored for the virtual machine allocation determination process (S404). The processing of S410 is performed.

For example, the flow management unit 202 stores the record of the analysis flow in which all the execution states 623 of the processing state management table 400 are “executable” (there is no “execution” section) as the target flow.

On the other hand, when there is an analysis flow in which the predicted time is not predicted (S403: NO), the flow management unit 202 equalizes the number or priority of the virtual machines 103 that execute the analysis flow among the analysis flows. This is stored for the virtual machine allocation determination process, and all the analysis flows enabled in S401 are stored as target flows (S405), and then the process of S410 is performed.

In S410, the flow management unit 202 first selects one of the target flows determined to be executed in S404 or S405, and determines whether or not there are a plurality of executable section flows in the target flow (S406). ). Specifically, for example, the flow management unit 202 refers to the execution state 623 of the record of each section in the target flow selected above in the processing state management table 400, and the record in which "executable" is stored. Check the number.

When there are a plurality of executable section flows in the target flow selected above (S406: YES), the flow management unit 202 determines the earliest section flow (for example, the time of the target data to be executed first). Only the earliest section flow) is set as the target section flow in the target flow selected above (S407). On the other hand, when there is only one executable section in the target flow selected above (S406: NO), the flow management unit 202 sets the one section as the target section flow in the target flow selected above. (S408).

By performing such processing, when some target flows have more executable section flows than other target flows, many section flows are assigned to some target flows, and each target flows. It is possible to prevent the parallel processing from being unable to be leveled between the flows.

The flow management unit 202 repeats the above processes of S406, S407, and S408 for all the target flows. This completes the flow execution target determination process (S409).

<Virtual machine allocation judgment processing>
FIG. 9 is a flowchart showing details of the virtual machine allocation determination process. First, the machine control unit 203 determines whether or not there is a virtual machine 103 to which each target section flow determined in the flow execution target determination process can be assigned (S501). Specifically, for example, the machine control unit 203 determines by referring to the allocated VM number 713 and the maximum VM allottable number 712 of each record in the VM management table 500.

If there is no virtual machine 103 to which the target section flow can be assigned (S501: NO), the machine control unit 203 ends the virtual machine allocation determination process (S507). On the other hand, if there is a virtual machine 103 to which the target section flow can be assigned (S501: YES), the process of S502 is performed.

In S502, the machine control unit 203 confirms whether or not the number of target flows (hereinafter referred to as the number of target flows) determined in the flow execution target determination process is 1 or greater than 1 (2 or more) (S502). When the number of target flows is 1 (S502: NO), the machine control unit 203 stores (determines) that the processing of the target flows is assigned to all of the assignable virtual machines 103 specified in S501 (determines). S503), the virtual machine allocation determination process ends (S507).

On the other hand, when the number of target flows is 2 or more (S502: YES), the machine control unit 203 executes the process of S504.

That is, in S504, when the fact that the analysis flow is executed by weighting is stored in the flow execution target determination process (S504: weighting), the machine control unit 203 is based on the predicted time of the target section flow in each target flow. , The virtual machine 103 to which the target section flow in each target flow is assigned is determined so that the end times of the target section flows in each target flow are substantially the same (S505). After that, the virtual machine allocation determination process ends (S507).

For this allocation, for example, the machine control unit 203 calculates the reciprocal of the predicted time for the target section flow in each target flow, and assigns the number of virtual machines 103 to which each target section flow is allocated at a ratio corresponding to each reciprocal. It is executed by a method such as determining. If an appropriate number cannot be calculated (for example, if the number as a natural number cannot be calculated), the machine control unit 203 assigns each virtual machine 103 a processing priority (for example, the amount of CPU or memory resource allocation). By setting, the target section flow in each target flow is assigned.

On the other hand, when it is stored in the flow execution target determination process that the analysis flow is to be executed evenly (S504: equal allocation), the machine control unit 203 executes the processing of the target section flow in each target flow. The virtual machines 103 to which each target flow (target section flow) is assigned are determined so that the number of virtual machines 103 is substantially the same as each other (S506). After that, the virtual machine allocation determination process ends (S507).

By the virtual machine allocation determination process as described above, a plurality of target section flows can be executed in parallel at the same time, and the time required for executing the plurality of target section flows can be adjusted to the same time. This makes it possible to prevent only a part of the analysis flow from being continuously executed for a long time.

The progress or result of the distributed processing executed as described above is displayed on the user operation terminal 105 or the like.

<Display example by user operation terminal 105>
FIG. 10 is an example of a screen displayed by the user operation terminal 105 showing the progress or result of the analysis process. As shown in the figure, the display screen 1000 displays a table 1010 showing the execution state or the execution result of the analysis process. In this table 1010, the current processing status 1011 (or its processing result) of each section flow in each analysis flow is displayed in chronological order for each virtual machine 103 in the virtual machine execution server 102. Then, the processing status 1011 also displays the amount of I / O generated by the processing. In addition, the total I / O amount 1012 generated by the analysis process is displayed for each time zone (for each data section).

In this way, the information on the result of the analysis flow by each virtual machine 103 or the information on the input / output amount of the data generated by the analysis flow is output to the display screen 1000, so that the user can analyze by the distributed processing system 100. It can be confirmed that the flows are sequentially processed in parallel, the load related to the I / O is distributed by this, and the distributed processing is stably executed.

As described above, according to the distributed processing system 100 of the present embodiment, the processing order of the processing unit (analysis flow) for the partition data of the target data is stored, and each partition of the target data performed by each processing unit (analysis flow). The load on the information processing device (virtual machine 103) due to data processing is calculated for each data partition, and the current execution state of each processing unit, the processing order of each processing unit, and the calculated load for each processing unit are calculated. Based on the above, a combination of a processing unit to be executed in parallel and an information processing device that executes the processing unit is determined, and each information processing device is made to execute these in parallel. As described above, the distributed processing system 100 of the present embodiment can allocate the processing of the analysis flow to each virtual machine 103 in units of data partitions according to the processing load related to each data partition of the target data. As a result, the processing of the target data can be distributed and processed in each virtual machine 103 for each data partition, and the target data can be stably distributed and processed.

A specific explanation of this effect is as follows.

First, FIG. 11 is a diagram showing an example of time-series changes in the usage status of the hardware resources of the virtual machine execution server 102 and the data lake 104 when the analysis process is executed in the conventional distributed processing system.

In this analysis process, analysis process B (analysis flow) is executed using the intermediate data generated by analysis process A, which is an analysis flow, and analysis process C (analysis flow) is performed using the intermediate data generated by analysis process B. It executes and outputs the final output data. That is, after the execution of the analysis process A is completed, the analysis process B is executed, and after the execution is completed, the analysis process C is executed.

In this case, the time-series changes in the processing allocation to each virtual machine 103 are as follows. First, all the virtual machines 103 execute the section A of the analysis process A (reference numeral 801), and after the processing is completed, the process of the next section (section B of the analysis process A) is executed (reference numeral 802). Then, after the execution of the processing of all the sections of the analysis process A is completed, the processing of the section A of the analysis process B, which is the next analysis flow, is executed (reference numeral 803), and then the processing of the entire section of the analysis flow of the analysis process B is executed. Execution of processing is completed. Then, the processing of the section A of the analysis processing C, which is the next analysis flow, is executed (reference numeral 804), and then the processing of the entire section of the analysis processing C is executed.

When such a process is executed, the unit time of the data to the data lake 104 during the execution of the analysis process B is compared with the amount of input / output 811 of the data to the data lake 104 during the execution of the analysis process A. Since the read / write amount per 812 is smaller, the usage amount of the resource related to the I / O (Input / Output) of the data lake has a margin in the analysis process B, and the resource related to the I / O becomes surplus.

On the other hand, the read / write amount 813 per unit time to the data lake during the execution of the analysis process C is larger than the read / write amount 812 per unit time of the data to the data lake 104 during the execution of the analysis process B. A delay may occur in reading and writing to the data lake 104 in the analysis process C, which may increase the execution time of the analysis process. That is, the resources related to I / O are insufficient.

In this way, in the conventional distributed processing system, processing is performed simply based on the execution order between each analysis flow, and as a result, the amount of I / O of each analysis flow changes greatly depending on the type of processing, and the processing is performed at that time. The above delay may occur and the efficiency of the analysis process may be reduced. In particular, if the difference in the amount of I / O between each analysis flow is significantly different, an unexpectedly large bottleneck will occur when the analysis flow to be executed is switched, and the processing will be significantly delayed, resulting in processing stability and reliability. Will hurt.

On the other hand, FIG. 12 is a diagram showing an example of time-series changes in the hardware resource usage status of the virtual machine execution server 102 and the data lake 104 when the analysis process is executed in the distributed processing system 100 of the present embodiment.

The distributed processing system 100 of the present embodiment determines the target flow and the target section flow each time the data processing of each section is completed, and performs these processes at an appropriate allocation ratio (for example, the processing in each section is performed). Assign it to the virtual machine 103 (so that it ends approximately at the same time) (S301 and S303 in FIG. 7).

That is, in the flow execution target determination process performed first, the target flow is determined to be only the analysis process A, and further, only the section flow corresponding to the first data partition is determined to be the target section flow ( In S407) of FIG. 8, only the section A of the analysis process A is executed (reference numeral 901).

Then, in the flow execution target determination process to be performed next, the target flow is first determined to be a plurality of analysis flows of the analysis process A and the analysis process B, and further, the predicted time of the analysis process B is not calculated, so that the flow is equal. The result determined to be allocation (S405 in FIG. 8), the process of the section B of the analysis process A, and the process of the section A of the analysis process B are evenly assigned to each virtual machine 103 (of the same number), and these processes. Are executed in parallel (reference numeral 902, reference numeral 903).

Further, in the flow execution target determination process to be performed next, first, the target flow is analyzed process C in addition to the analysis process A (process of section B) and analysis process B (process of section A). As a result of determining that the target section flow of the analysis process C is the section A, the process of the section B of the analysis process A, the process of the section A of the analysis process B, and the process of the section A of the analysis process C. Are executed in parallel (reference numeral 904).

In this way, by executing the processing of each section in the analysis flow in parallel, the processing related to the analysis flow is distributed not only in the analysis flow unit but also in the section unit. Therefore, the conventional distributed processing system described above. The analysis process is more stably distributed, and the load related to the process is also distributed. That is, not only stabilization by distribution of resources but also stabilization by distribution of processing in the time axis direction is realized.

Further, in the distributed processing system 100 of the present embodiment, the analysis flow is divided into a plurality of section flows according to the time series (data partition) of the data to be processed, so that only the load related to the arithmetic processing of the CPU or the like is required. However, the load related to the input / output process (I / O) is also distributed in the time axis direction.

For example, since only the analysis process A is executed in the first flow execution target determination process, the amount of data read / written to the data lake 104 per unit time related to the execution of the analysis process A 911 (hereinafter, unit I). (Referred to as / O amount) occurs (reference numeral 911).

Then, in the second flow execution target determination process, an intermediate unit I / O amount is generated between the case where the analysis process A is executed independently and the case where the analysis process B is executed alone. (Code 912). Further, in the third flow execution target determination process, the case where the analysis process A is executed independently, the case where the analysis process B is executed independently, and the case where the analysis process C is executed independently. An intermediate unit I / O amount of the three parties will be generated (reference numeral 913).

As described above, according to the distributed processing system 100 of the present embodiment, the amount of data read / written to / from the data lake 104 per unit time is more detailed in the time axis direction as compared with the case of the above-mentioned ordinary distributed processing system. Is leveled to. As a result, for example, reading and writing data to and from the data lake 104 can be prevented from becoming a bottleneck in the analysis process, and the analysis process can be performed stably.

That is, according to the distributed processing system 100 of the present embodiment, as a resource in the analysis processing, not only the load related to the processor but also the load related to I / O is leveled, and as a result, the entire processing load in the distributed processing system 100 is equalized. It can be stabilized.

In addition to the above effects, the distributed processing system 100 of the present embodiment stores the constraint conditions regarding the parallel execution of the processing unit (analysis flow) by the information processing device (virtual machine 103) (VM management table 500). Since the virtual machine 103 that satisfies the constraint condition is determined as the virtual machine 103 that executes the processing unit in parallel, the distributed processing and the data processing by the virtual machine 103 are stably performed within the constraints of the unavoidable virtual machine 103. It can be carried out.

For example, by storing the maximum number of virtual machines 103 capable of executing analysis flow processing in parallel as a constraint condition, for example, the hardware or software specifications of the virtual machine execution server 102 and the virtual machine 103 are satisfied. Stable distributed processing and data processing can be performed.

Further, the distributed processing system 100 of the present embodiment calculates the estimated time for executing the processing of the processing unit (analysis flow) for each processing unit as a load on the information processing device (virtual machine execution server 102 or virtual machine 103). Then, based on this estimated time, the distribution of each information processing device (processing priority, etc.) for executing the processing of the processing unit in parallel is determined, so that the actual virtual machine execution server 102 or virtual machine 103 is actually executed. It is possible to distribute rational processing based on the operation results. As a result, stable distributed processing and data processing can be performed.

Further, the distributed processing system 100 of the present embodiment executes each of the plurality of processing units (analysis flow) executed in parallel when there is a processing unit for which the predicted time has not been calculated. Since the number of information processing devices (virtual machines 103) to be processed is equalized to each other, even if the operation record of a certain virtual machine 103 is small and the processing time of the analysis flow is unknown or inaccurate, it is reasonably distributed. Processing can be performed.

Further, in the distributed processing system 100 of the present embodiment, when determining the processing unit (analysis flow) to be executed in parallel, if there are a plurality of data partitions that can be processed in the processing unit, the processing order is the highest. Since only the data of the previous partition is processed, even if the number of executable section flows differs between the analysis flows, only some analysis flows are executed in a biased manner, and parallel processing can be leveled. It can be prevented from disappearing.

The above description of the embodiment is for facilitating the understanding of the present invention, and does not limit the present invention. The present invention can be modified and improved without departing from the spirit of the present invention, and the present invention includes its equivalents.

For example, in the present embodiment, the main body that performs the analysis process is a plurality of virtual machines 103, but a plurality of physical servers may perform these analysis processes. Since the distributed processing system 100 of the present embodiment can support distributed processing by any type of information processing apparatus, hardware resources and software resources can be efficiently used.

For example, in the present embodiment, a table (database) is shown as a data storage format, but the data storage format is not limited to the database and may be any method.

Further, in the present embodiment, it is assumed that there are a plurality of virtual machine execution servers 102, but only one virtual machine execution server 102 may be provided, and a plurality of virtual machines 103 may be provided in the virtual machine execution server 102. ..

100 Distributed processing system, 101 Virtual machine management server, 102 Virtual machine execution server, 103 Virtual machine, 104 Data lake, 211 Flow table storage unit, 201 Processing load calculation unit, 202 Flow management unit, 203 Machine control unit

Claims (10)

Is configured to include a plurality of virtual machines, for a given process in which a plurality comprises a processing flow for processing predetermined data comprising a plurality of compartments in each of the compartments, the processing flow to at least one of the plurality of virtual machines A distributed processing system equipped with a processor and a memory capable of executing the processing flow in parallel by allocating.
A flow table storage unit that stores the execution order of the plurality of processing flows for the data in the partition, and
A processing load calculation unit that calculates the load on the virtual machine by processing the data of the partition according to the processing flow for each partition,
The processing flow executed in parallel based on the current execution state of each processing flow , the execution order of each processing flow , and the load calculated for each processing flow , and the virtual machine that executes the processing flow. The flow management department that decides the combination of machines and
A machine control unit that causes each virtual machine to execute parallel processing indicated by the determined combination.
Equipped with a,
The processing load calculation unit calculates the estimated time for executing the processing flow as the load for each of the processing flows.
The flow management unit
When there are a plurality of the processing flows executed in parallel, the priority regarding the virtual machine or its allocation to be assigned to each of the plurality of processing flows is determined based on the calculated estimated time.
When there is the processing flow for which the predicted time has not been calculated among the plurality of processing flows executed in parallel, the number of the virtual machines executing each of the plurality of processing flows is made equal to each other. ,
Distributed processing system. A machine management table storage unit that stores constraints related to parallel execution of the processing flow by the virtual machine is provided.
The flow management section, the constraint satisfies the virtual machine is determined as the virtual machine running the process flow in parallel,
The distributed processing system according to claim 1. As the constraint condition, the machine management table storage unit stores the maximum number of virtual machines capable of executing the processing flow in parallel.
The flow management unit determines the number of virtual machines satisfying the maximum number indicated by the constraint condition as the virtual machines that execute the processing flow in parallel.
The distributed processing system according to claim 2. When the flow management unit determines the processing flows to be executed in parallel, if there are a plurality of sections of the data that can be processed in the processing flow, the predetermined first processing is performed. The distributed processing system according to claim 1, wherein only the data partition is determined to be processed in the processing flow. The distributed processing system according to claim 1, further comprising an output unit that outputs information regarding the result of the processing flow executed by each virtual machine or the amount of data input / output generated by the processing flow. A machine management table storage unit that stores constraints related to parallel execution of the processing flow by the virtual machine, and
Further provided with an output unit that outputs information regarding the result of the processing flow executed by each of the virtual machines or the amount of data input / output generated by the processing flow.
As the constraint condition, the machine management table storage unit stores the maximum number of virtual machines capable of executing the processing flow in parallel.
The flow management unit
The number of the virtual machines satisfying the maximum number indicated by the constraint condition is determined as the virtual machines that execute the processing flow in parallel.
When determining the processing flow to be executed in parallel, if there are a plurality of data partitions that can be processed in the processing flow, only the predetermined data partition that is processed first is said. Determine to be processed in the processing flow,
The distributed processing system according to claim 1. Is configured to include a plurality of virtual machines, for a given process in which a plurality comprises a processing flow for processing predetermined data comprising a plurality of compartments in each of the compartments, the processing flow to at least one of the plurality of virtual machines It is a distributed processing method in a distributed processing system that can execute the processing flow in parallel by allocating.
A machine management server with a processor and memory
A flow table storage process that stores the execution order of the plurality of processing flows for the data in the partition, and
A processing load calculation process that calculates the load on the virtual machine by processing the data of the partition according to the processing flow for each partition, and
The processing flow executed in parallel based on the current execution state of each processing flow , the execution order of each processing flow , and the load calculated for each processing flow , and the virtual machine that executes the processing flow. Flow management process that determines the combination of machines and
Machine control processing that causes each virtual machine to execute parallel processing indicated by the determined combination, and
The execution,
In the processing load calculation process, as the load, the estimated time for executing the processing flow is calculated for each processing flow.
In the flow management process
When there are a plurality of the processing flows executed in parallel, the priority regarding the virtual machine or its allocation to be assigned to each of the plurality of processing flows is determined based on the calculated estimated time.
When there is the processing flow for which the predicted time has not been calculated among the plurality of processing flows executed in parallel, the number of the virtual machines executing each of the plurality of processing flows is made equal to each other. ,
Distributed processing method. The machine management server further executes machine management table storage processing that stores constraints related to parallel execution of the processing flow by the virtual machine.
The flow management process, the constraints satisfying the virtual machine is determined as the virtual machine running the process flow in parallel,
The distributed processing method according to claim 7. Is configured to include a plurality of virtual machines, for a given process in which a plurality comprises a processing flow for processing predetermined data comprising a plurality of compartments in each of the compartments, the processing flow to at least one of the plurality of virtual machines To a distributed processing system equipped with a processor and memory that can execute the processing flow in parallel by allocating
A flow table storage process that stores the execution order of the plurality of processing flows for the data in the partition, and
A processing load calculation process that calculates the load on the virtual machine by processing the data of the partition according to the processing flow for each partition, and
The processing flow executed in parallel based on the current execution state of each processing flow , the execution order of each processing flow , and the load calculated for each processing flow , and the virtual machine that executes the processing flow. Flow management process that determines the combination of machines and
Machine control processing that causes each virtual machine to execute parallel processing indicated by the determined combination, and
To run ,
In the processing load calculation process, as the load, the estimated time for executing the processing flow is calculated for each processing flow.
In the flow management process
When there are a plurality of the processing flows executed in parallel, the priority regarding the virtual machine or its allocation to be assigned to each of the plurality of processing flows is determined based on the calculated estimated time.
When there is the processing flow for which the predicted time has not been calculated among the plurality of processing flows executed in parallel, the number of the virtual machines executing each of the plurality of processing flows is made equal to each other. ,
Distributed processing program. Further execute the machine management table storage process that stores the constraint conditions related to the parallel execution of the process flow by the virtual machine.
The flow management process, the constraints satisfying the virtual machine is determined as the virtual machine running the process flow in parallel,
The distributed processing program according to claim 9.

Images