JP2010218307A - Distributed calculation controller and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the following problem: because a conventional Map/Reduce type distributed calculation processing system is configured by assuming a physical computer, a distributed arrangement method of calculation processing under environment using system virtualization technology or a communication method between virtual machines are not considered, so that deterioration of calculation processing performance occurs. <P>SOLUTION: In this distributed calculation controller and method for distributedly arranging a plurality of pieces of second calculation processing constituting first calculation processing in the plurality of virtual machines operating on one or more physical machines, various kinds of arrangement patterns wherein the plurality of pieces of the second calculation processing are distributedly arranged in the plurality of virtual machines are detected, cost of each arrangement pattern is calculated, the arrangement pattern wherein the cost becomes minimum is selected based on a calculation result of the calculation, and the plurality of pieces of the second calculation processing are distributedly arranged in the plurality of virtual machines according to the arrangement pattern. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a distributed calculation control apparatus and method, and is suitable for application to a distributed calculation system that performs distributed calculation processing using a system virtualization function.

  In recent years, processing demands for large-scale and large-scale data such as Web data, input data from sensors, audio data, and moving image data are increasing. As a processing system for processing such a large amount of large-scale data, there is a Map / Reduce type distributed computing system (see Non-Patent Document 1).

  In a map / reduce type distributed calculation system, a virtual distributed file system is provided on a calculation node, and one calculation process is distributed among a plurality of calculation nodes. At this time, in such a distributed calculation system, the target calculation process is defined by being divided into two calculation processes of a map process and a reduce process. The Map process is a process of converting a data set represented by a set of <key, value> into a set of another set of <key ', value'>. The Reduce process is a process for counting the processing results of the Map process.

  Input / output data between Map processing and Reduce processing is exchanged through a distributed file system constructed on a distributed computing system. An operation on the distributed file system is performed by communication between the management program of the distributed file system and a data processing program that executes Map / Reduce processing. In the Map / Reduce type distributed calculation processing, data on the distributed file system is divided into units called data blocks, and Map processing and Reduce processing are sequentially applied to these data blocks.

  According to such a map / reduce type distributed calculation system, a series of calculation processes for each data block can be simultaneously distributed and executed by a plurality of calculation nodes. Arrangement of processes for each computation node is basically performed by assigning processes to the computation nodes in the order of completion of the processes.

On the other hand, in recent years, a technique for operating a plurality of virtual machines on a physical machine (hereinafter referred to as a physical machine) is known. Such a virtualization technique is called a system virtualization technique, and a virtual machine that operates on a physical machine is called a virtual machine. A virtual machine looks like a physical machine to software running on the virtual machine.
J. Dean et al., “MapReduce: Simplified Data Processing on Large Clusters”, In Proceedings of OSDI'04, 2004

  A conventional Map / Reduce type distributed computation system assumes that each computation node that executes Map processing and Reduce processing is a physical machine. For this reason, in the conventional Map / Reduce type distributed computing system, no consideration is given to the distributed arrangement method of calculation processing in an environment using system virtualization technology and the communication method between virtual machines. As a result, in the conventional map / reduce type distributed computing system, there has been a situation in which the computing performance of the entire system deteriorates in an environment using system virtualization technology.

  The present invention has been made in consideration of the above points, and an object of the present invention is to propose a distributed calculation control apparatus and method capable of improving the calculation processing performance in an environment using a system virtualization technique.

  In order to solve such a problem, in the present invention, a distributed calculation control device that distributes and arranges a plurality of second calculation processes constituting the first calculation process in a plurality of virtual machines operating on one or a plurality of physical machines. The arrangement pattern detection unit for detecting various arrangement patterns for distributing and arranging the plurality of second calculation processes in the plurality of virtual machines, the cost calculation unit for calculating the cost for each arrangement pattern, and the cost calculation A distribution arrangement unit that selects an arrangement pattern that minimizes the cost based on the calculation results of the units, and distributes the plurality of second calculation processes to the plurality of virtual machines according to the arrangement pattern. And

  According to the present invention, in the distributed calculation control method for distributing and arranging a plurality of second calculation processes constituting the first calculation process in a plurality of virtual machines operating on one or a plurality of physical machines, Based on a first step of detecting various arrangement patterns in which the second calculation process is distributed and arranged in the plurality of virtual machines, a second step of calculating a cost for each of the arrangement patterns, and a calculation result of the calculation And a third step of selecting an arrangement pattern with the lowest cost and distributing the plurality of second calculation processes to the plurality of virtual machines according to the arrangement pattern.

  ADVANTAGE OF THE INVENTION According to this invention, the calculation processing performance in the environment using a system virtualization technique can be improved.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) 1st Embodiment In FIG. 1, 100 shows the distributed calculation system by this embodiment as a whole. The distributed calculation system 100 includes a distributed calculation control node 101, first and second distributed calculation nodes 102 and 103, and a network 104. Although FIG. 1 illustrates the case where there are two distributed calculation nodes (first and second distributed calculation nodes 102 and 103), there are three or more distributed calculation nodes in the distributed calculation system 100. Even in this case, the present invention can be applied.

  The distributed computation control node 101 includes hardware 110 and an external storage device 111. The hardware 110 includes information processing resources such as a CPU (Central Processing Unit) 110 </ b> A and a shared memory, and the distributed calculation control system 112 operates on the hardware 110.

  The distributed calculation control system 112 is a function realized by the CPU 110 </ b> A constituting the hardware 110 executing a corresponding program stored in the external storage device 111, and the distributed calculation definition 113 stored in the external storage device 111. Is distributed based on the distributed calculation definition 113 for the first and second distributed calculation nodes 102 and 103. The distributed computation control system 112 includes a job placement processing unit 114 that is a function for executing job placement processing that is characteristic processing of the present distributed computation system 100.

  The first and second distributed computation nodes 102 and 103 include hardware 120 and 130 and external storage devices 121 and 131. The hardware 120 and 130 includes information processing resources such as CPUs (Central Processing Units) 120A and 130A and shared memories 120B and 130B, and the system virtualization functions 122 and 132 operate on the hardware 120 and 130. .

  The system virtualization functions 122 and 132 are realized by the CPUs 120A and 130A constituting the hardware 120 and 130 executing corresponding programs stored in the external storage devices 121 and 131 in the same distributed computing nodes 102 and 103. It is a function. Then, virtual machines 123 and 124 operate on the system virtualization function 122 of the first distributed computing node 102, and distributed processing applications 125 and 126 operate on these virtual machines 123 and 124. In addition, virtual machines 134 and 135 operate on the system virtualization function 132 of the second distributed computing node 103, and distributed processing applications 136 and 136 operate on these virtual machines 134 and 135. 1 illustrates the case where the number of virtual machines operating on the system virtualization functions 122 and 132 is two, the number of virtual machines operating on the system virtualization functions 122 and 132 is illustrated. The present invention can be applied even when the number is three or more.

  Between the virtual machines 123, 124 or 134, 135 operating on the same distributed computing node (first or second distributed computing node 102, 103), the first or second distributed computing node 102, 103 Data can be transferred at high speed by the communication paths 127 and 137 using the shared memories 120B and 130B. On the other hand, data is transferred between the virtual machines 123, 124 or 134, 135 operating on another distributed computing node via a slower communication path 140 via the network 104.

  FIG. 2 shows a configuration example of a conventional distributed computing system 200. The distributed calculation system 200 is configured by connecting a distributed calculation control node 201 and first to fourth distributed calculation nodes 202 to 205 via a network 206.

  The distributed calculation control node 201 includes hardware 210 and an external storage device 211, and the distributed calculation control system 212 operates on the hardware 210. The distributed calculation control system 212 is a function realized by the CPU constituting the hardware 210 executing a corresponding program stored in the external storage device 211, and the distributed calculation definition 213 stored in the external storage device 211. Is distributed and calculation processing is distributed to the first to fourth distributed calculation nodes 202 to 205.

  The first to fourth distributed computation nodes 202 to 205 are configured by hardware 220, 230, 240, 250 and external storage devices 221, 231, 241, 251 respectively. The hardware 220, 230, 240, 250 includes information processing resources such as a CPU, a memory, and a communication interface, and the distributed computing applications 222, 232, 242 on these hardware 220, 230, 240, 250, respectively. 252 operates. Data transfer between the distributed calculation applications 222, 232, 242, and 252 is performed via a communication path 260 using a relatively low-speed network 206.

  FIG. 3 shows the definition of the distributed calculation process targeted by the present invention. The distributed calculation process includes a map process 301 and a reduce process 302. The map processing 301 inputs a plurality of <key, value> pairs from the data blocks of the distributed file system stored in the external storage devices 121 and 131 on the respective distributed computing nodes 102 and 103, and <key, 1>. Output a pair of In Reduce processing, the number of occurrences of each key is totaled and <key, total value> is output. In the following, each Map process 301 and Reduce process will be appropriately referred to as a unit calculation process.

  FIG. 5 shows an arrangement example in which such a series of calculation processing is distributed in the conventional distributed calculation system 200 described above with reference to FIG. In the example of FIG. 5, the Map processing 502 and / or the Reduce processing 503 for the individual data 501 are executed in the hardware 220, 230, 240, 250 in the first to fourth distributed computing nodes 202-205, and the final The result data 504 is output.

  In this case, since the first to fourth distributed computation nodes 202 to 205 are independent from each other, data transfer between the hardware 220, 230, 240, and 250 is performed via the low-speed communication path 260 using the network 206. Need to do through. For example, in FIG. 5, the processing result of the MAP process 503 “MAP # 1” for the data 501 “BLK # 1” and the processing result of the MAP process 503 “MAP # 2” for the data 501 “BLK # 2” Is used in the Reduce process “RED # 1”, the hardware (hardware 220, 230, 240) that executes the Reduce process “RED # 1” from the processing result of the MAP process 503 “MAP # 2”. , 250).

  On the other hand, FIGS. 4A and 4B show an arrangement example when the series of calculation processing shown in FIG. 3 is distributed in the distributed calculation system 100 (FIG. 1) according to the present embodiment.

  In FIG. 4A, two Map processes 402 “MAP # 1” and “MAP # 2” and a subsequent Reduce process 403 “RED # 1” are executed on the same hardware 120. A pattern (pattern # 1) in which two Map processes 403 "MAP # 3" and "MAP # 4" and a subsequent Reduce process 404 "RED # 2" are executed on the same hardware 130 is shown. .

  4B shows that two Map processes 403 “MAP # 1” and “MAP # 2” and a subsequent Reduce process 404 “RED # 1” are performed on different hardware 120 and 130. A pattern (pattern that executes two Map processes 403 “MAP # 3” and “MAP # 4” and a subsequent Reduce process 404 “RED # 2” on different hardware 130 and 120. # 2).

  In the pattern (pattern # 1) in FIG. 4A, the data transfer when the processing result of the map process 403 is used in the reduce process 404 can be performed via the high-speed communication path 127 (FIG. 1). On the other hand, in the pattern (pattern # 2) of FIG. 4B, it is necessary to perform data transfer when using the processing result of the Map process 403 in the Reduce process 404 via the low-speed communication path 140 (FIG. 1). It becomes.

  FIG. 6 shows a specific configuration of the distributed calculation control system 112 of FIG. As shown in FIG. 6, the distributed calculation control system 112 includes a job control unit 601 that controls the entire distributed calculation processing, a task control unit 602 that controls individual unit calculation processing, and an individual on the distributed file system. A data management unit 603 that manages data and a metadata management unit 604 that manages logical data of the distributed file system are configured.

  A plurality of task control units 602 and data management units 603 exist on the first and second distributed computation nodes 102 and 103. The job control unit 601 includes a job placement processing unit 114 that executes job placement processing that is characteristic processing of the distributed computing system 100. The job placement processing unit 114 receives the distributed calculation definition 113 as an input, calculates which virtual machine the individual distributed calculation processing is placed in, and controls the task control unit 602 based on the calculation result.

  FIG. 7 shows a processing procedure of job placement processing executed by the job placement processing unit 114. The job placement processing unit 114 starts the job placement processing shown in FIG. 7 in response to, for example, a user operation. First, a flow graph of calculation processing input by the user (hereinafter referred to as a processing flow graph G). ) And a set of virtual machines available at that time (hereinafter referred to as a virtual machine set M) (SP1).

  The processing flow graph G can be obtained by existing compiler mounting technology. By the processing in step SP1, for example, a processing flow graph G as shown in FIGS. 4A and 4B can be obtained. The virtual machine set M can be obtained by collecting information on all virtual machines existing in the distributed computing system 100 and extracting available virtual machines from these virtual machines.

  Subsequently, the job arrangement processing unit 114 randomly arranges each element (unit calculation process) of the processing flow graph G in any of the virtual machines belonging to the virtual machine set M (hereinafter referred to as an arrangement pattern q). (Referred to as an arrangement pattern set Q hereinafter) (SP2). Then, the job arrangement processing unit 114 stores the information of the arrangement pattern set Q obtained in this way in a memory (not shown) in the hardware 110 (FIG. 1).

  FIG. 8 shows an arrangement pattern set Q when the series of calculation processes shown in FIG. 4 is executed by the distributed calculation system 100 of FIG. In this arrangement pattern set Q, four Map processes “MAP # 1” to “MAP # 4” and two Reduce processes “RED # 1” and “RED # 2” are respectively referred to as “VM # 1”. ”To“ VM # 4 ”, the entire arrangement pattern q in the case of distributed arrangement in the four virtual machines 123, 124, 133, and 134 is included.

  In FIG. 8, each row represents an individual arrangement pattern q. For example, an arrangement pattern q having an item number “1” represents all unit calculation processes (all four Map processes and two Reduce processes) as “ “VM # 1” represents an arrangement pattern arranged in the virtual machine 123, and an arrangement pattern q having an item number “2” is obtained by performing four Map processes and one Reduce process (“RED # 1”) on “VM # 1”. Represents a placement pattern in which another Reduce process (“RED # 2”) is placed in the virtual machine 124 called “VM # 2”.

  Next, the job placement processing unit 114 calculates the total cost value Cq of each placement pattern q constituting the placement pattern set Q described above (SP3). Furthermore, the job arrangement processing unit 114 selects an arrangement pattern q that minimizes the total cost value Cq (SP4), and thereafter ends the job arrangement process.

  FIG. 9 shows specific processing contents of the job placement processing unit 114 in step SP3 of the job placement processing described above. When the job placement processing unit 114 proceeds to step SP3 of the job placement processing, the job placement processing portion 114 starts the cost calculation processing shown in FIG. 9, and first reads out information of the placement pattern set Q obtained in step SP2 of the job placement processing from the memory. (SP10).

  Subsequently, the job arrangement processing unit 114 determines whether or not the arrangement pattern set Q is an empty set (SP11). If the arrangement pattern set Q is an empty set (SP11: YES), this cost calculation process is performed. To return to the job placement process.

  On the other hand, when the arrangement pattern set Q is not an empty set (SP11: NO), the job arrangement processing unit 114 extracts one element (arrangement pattern q) from the arrangement pattern set Q, and determines the arrangement pattern q. Delete from the arrangement pattern set Q. Further, the job placement processing unit 114 obtains a set of virtual machines to which each unit calculation process in the placement pattern q is assigned (hereinafter referred to as an assigned virtual machine set P). Furthermore, the job placement processing unit 114 initializes the above-described total cost value Cq to “0” (SP12).

  Subsequently, the job placement processing unit 114 determines whether or not the assigned virtual machine set P is an empty set (SP13). If the assigned virtual machine set P is not an empty set (SP13: NO), the job placement processing unit 114 selects one element p (virtual machine to which unit calculation processing is assigned) from the assigned virtual machine set P. Then, the selected virtual machine is removed from the assigned virtual machine set P. The job placement processing unit 114 calculates the cost Cp related to the selected virtual machine, and adds the obtained cost Cp to the total cost value Cq (SP14).

  Subsequently, the job placement processing unit 114 returns to step SP13, and thereafter repeats the same processing until the assigned virtual machine set P of the target placement pattern q becomes an empty set (SP13-SP14-SP13). .

  Then, the job placement processing unit 114 eventually obtains an affirmative result in step SP13 by obtaining the total cost value Cq obtained by summing all the costs Cp for the virtual machines included in the assigned virtual machine set P of the placement pattern q that is the target at that time. If obtained (SP13: YES), after storing the cost total value Cq of the arrangement pattern q (SP15), the process returns to step SP11.

  The job placement processing unit 114 thereafter repeats the same processing until a positive result is obtained in step SP11 (SP11 to SP15). When the job placement processing unit 114 eventually obtains a positive result in step SP11 by completing the same processing for all the placement patterns q constituting the placement pattern set Q acquired in step SP10, it ends this cost calculation processing. .

  It should be noted that conditions can be set as appropriate according to the target to be optimized during the cost calculation process related to each unit calculation process in step SP15 of the above-described cost calculation process. As such conditions, it is conceivable to apply execution time, power consumption, and / or I / O amount. An example of such conditions is shown in FIG. When the condition shown in the condition column 1000 is matched, the cost when the process is assigned to the virtual machine is calculated as the cost shown in the cost column 1001.

  In this case, in the present embodiment, for the purpose of allocating the Map process and the Reduce process using the processing result of the Map process to the virtual machines operating on the same hardware 120 and 130, The cost when the Map processing and Reduce processing having the above relationship are distributed and arranged in one or a plurality of virtual machines operating on the same hardware 120, 130 is set low (item number “1” in FIG. 10). The map processing and the Reduce processing having such a relationship are set high in cost when they are distributed and arranged in one or a plurality of virtual machines operating on different hardware 120 and 130 (item numbers in FIG. 10). (See “2”).

  As a result, since only the condition of item number “1” in FIG. 10 is applicable to the arrangement pattern in FIG. 4A, the cost of this arrangement pattern can be calculated as “1”. Further, since the conditions of item numbers “2” to “5” in FIG. 10 correspond to the arrangement pattern of FIG. 4B, the cost of this arrangement pattern can be calculated as “18”. Therefore, in the example of FIGS. 4A and 4B, the job placement processing unit 114 selects the placement pattern of FIG. 4A in step SP4 of FIG. Can be done.

  From the above viewpoint, in the case of the present embodiment, a condition table as shown in FIG. 10 in which the conditions at the time of cost calculation are registered is stored in the external storage device 111. The cost calculation process in step SP15 of the cost calculation process is executed with reference to FIG.

  FIG. 11 shows the processing contents when the execution time is a condition for the cost calculation processing for each element p of the allocation virtual machine set P of the allocation pattern q to be executed at that time in step SP15 of the cost calculation processing. An example is shown.

  When the job placement processing unit 114 proceeds to step SP15 of the job placement process described above with reference to FIG. 9, it starts the element cost calculation process shown in FIG. ”(SP20).

  Subsequently, the job placement processing unit 114 is a virtual machine (hereinafter referred to as a communication destination virtual machine) that is a communication destination of the element p (a virtual machine to which unit calculation processing is assigned) of the target virtual machine set P at that time. (Referred to as a communication destination set D hereinafter) (SP21), and determines whether the communication destination set D is an empty set (SP22).

  If the job placement processing unit 114 obtains a negative result in this determination (SP23), it extracts one element d (communication destination virtual machine) of the communication destination set D and deletes the communication destination virtual machine from the communication destination set D. To do. In addition, the job placement processing unit 114 communicates between the virtual machine to which the target unit calculation process is assigned (hereinafter referred to as a communication source virtual machine as appropriate) and the communication destination virtual machine. Is added to the total communication time pt (SP23).

  Subsequently, the job placement processing unit 114 returns to step SP22, and thereafter repeats the same processing until the communication destination set D for the target communication source virtual machine becomes an empty set (SP22-SP23-SP22). ).

  Then, the job placement processing unit 114 eventually obtains an affirmative result in step SP22 by obtaining the total communication time pt that is the total value of the communication times t between the communication source virtual machine and the communication destination virtual machines that are the target at that time. If obtained (SP22: YES), based on the total communication time pt acquired at that time and the condition described above with reference to FIG. 10 included in the distributed calculation definition 113, the element p of the assigned virtual machine set P that is the target at that time Cost Cp is acquired (SP24), and then the process returns to the cost calculation process (FIG. 9).

  Even when the cost calculation conditions are power consumption and I / O processing, it is possible to perform cost calculation using the same algorithm as in FIG.

  As described above, in the distributed calculation system 100 according to the present embodiment, all the arrangement patterns distributed to the virtual machines to which individual unit calculation processes can be allocated are obtained, the costs are calculated for each of the arrangement patterns p, and the cost is the highest. By selecting a low arrangement pattern p, the Map process and the Reduce process using the processing result of the Map process are operated on the same physical machine (first or second distributed computing node 102, 103). Can be distributed to virtual machines. As a result, the processing performance of the distributed calculation processing in the distributed calculation system 100 can be improved.

(2) Second Embodiment According to the first embodiment, Map processing and Reduce processing using the processing result of the Map processing can be distributed and arranged in virtual machines operating on the same physical machine. it can.

  In this case, data transfer using the high-speed communication path 127 (FIG. 1) using the shared memories 120B and 130B (FIG. 1) can be performed between the virtual machines operating on the same physical machine. When the processing result of the Map process is transferred to the Reduce process, the speed of the distributed calculation process can be increased by using the communication path 127 (FIG. 1) using the shared memories 120B and 130B.

  FIG. 12 shows the processing contents of processing for setting a communication path between virtual machines (hereinafter referred to as communication path setting processing) executed by the job placement processing unit 114 based on such a viewpoint. The job placement processing unit 114 executes the communication path setting process shown in FIG. 12 based on a corresponding program stored in the external storage device 111 (FIG. 1).

  That is, when the job placement processing unit 114 finishes the job placement processing described above with reference to FIG. 7, the job placement processing unit 114 starts the communication path setting processing shown in FIG. 12, and first assigns the placement pattern q selected in step SP4 of the job placement processing. A virtual machine set P is obtained (SP30).

  Subsequently, the job placement processing unit 114 determines whether or not the assigned virtual machine set P is an empty set (SP31). If the assigned virtual machine set P is not an empty set (SP31: NO), the assigned virtual machine set P is assigned. One element p (virtual machine to which unit calculation processing is assigned) of the virtual machine set P is selected, and its communication destination set D is obtained (SP32). In addition, the job placement processing unit 114 deletes the element p from the assigned virtual machine set P together with this.

  Subsequently, the job placement processing unit 114 determines whether or not the communication destination set D is an empty set (SP33), and if a negative result is obtained (SP33: NO), the element d (communication) of the communication destination set D is obtained. (Destination Virtual Machine) is selected (SP34), and whether or not the communication destination virtual machine and the virtual machine (communication source virtual machine) selected in Step SP32 operate on the same hardware 120, 130. Judgment is made (SP35). At this time, the job placement processing unit 114 deletes the element d from the communication destination set D.

  If the job placement processing unit 114 obtains a positive result (SP35: YES) in this determination, it sets the communication path 127 using the shared memories 120B and 130B as the communication path between the communication source virtual machine and the communication destination virtual machine. (SP36). Then, the job placement processing unit 114 returns to step SP33.

  On the other hand, when the job placement processing unit 114 obtains a negative result (SP35: NO) in the determination at step SP35, it uses the network 104 (FIG. 1) as a communication path between the communication source virtual machine and the communication destination virtual machine. The established communication path 140 (FIG. 1) is set (SP37). Then, the job placement processing unit 114 returns to step SP33.

  When the job placement processing unit 114 eventually obtains an affirmative result in step SP33 by completing similar processing for all elements d of the communication destination set D of the virtual machine (communication source virtual machine) selected in step SP32, step SP31 is obtained. Thereafter, the processing after step SP31 is similarly performed.

  Then, the job placement processing unit 114 eventually gives a positive result in step SP31 by completing the same processing for all the elements p of the assigned virtual machine set P to which the placement pattern q selected in step SP4 of the job placement processing (FIG. 7). If it is obtained (SP31: YES), this communication path setting process is terminated.

  As described above, according to the present embodiment, the communication path 127 using the shared memories 120B and 130B can be set as a communication path for data transfer between virtual machines operating on the same hardware 120 and 130. Therefore, it is possible to speed up the distributed calculation processing.

(3) Third Embodiment According to the first embodiment, Map processing and Reduce processing using the processing result of the Map processing can be distributed and arranged in virtual machines operating on the same physical machine. However, by executing the map process and the reduce process having such an input / output relationship in the same virtual machine, the speed of the distributed calculation process can be further increased.

  FIG. 13 is a flowchart illustrating a processing procedure of job summarization processing in which Map processing and Reduce processing having such an input / output relationship are collectively arranged in one virtual machine. The job placement processing unit 114 performs the job summarization processing shown in FIG. 13 after the job placement processing described above with reference to FIG. 7, for example, based on the program stored in the external storage device 111 (FIG. 1). This is executed before the communication path setting process described above with reference to FIG.

  That is, when the job placement processing unit 114 completes the job placement processing, the job placement processing unit 114 starts the job summarization processing. First, in the placement pattern p selected in step SP4 of the job placement processing, the map processing and the processing result of the map processing are performed. Is obtained (hereinafter referred to as a related processing set Y) (SP40), and it is determined whether or not the related processing set Y is an empty set (SP41).

  The job placement processing unit 114 selects one element y (a combination of the map process and the reduce process) of the related process set Y (SP42), and for the element y, the virtual machine that executes the map process and the reduce process are selected. It is determined whether or not the virtual machine to be executed operates on the same hardware 120 and 130 (SP43).

  If the job placement processing unit 114 obtains a negative result in this determination, the job placement processing unit 114 returns to step SP41. If the job placement processing unit 114 obtains a positive result, the job placement processing unit 114 determines whether the map processing and the reduce processing can be merged (SP44). ).

  If the job placement processing unit 114 obtains a negative result in this determination, it returns to step SP41, and if it obtains a positive result, it merges such Map processing and Reduce processing, and a series of Map / After the Reduce process is reassigned to one of the virtual machines to which the Map process or Reduce process has been assigned (SP45), the process returns to step SP41.

  Thereafter, the job placement processing unit 114 repeats the same processing until a positive result is obtained in step SP41 (SP41 to SP45-SP41). If the job placement processing unit eventually obtains a positive result in step SP41 by completing the same processing for all elements y belonging to the related processing set Y, it ends this job summarization processing.

  As described above, according to the present embodiment, Map processing and Reduce processing using the processing result of the Map processing can be collectively arranged in the same virtual machine, so the speed of distributed calculation processing can be further increased. Can be achieved.

(4) Other Embodiments In the above first to third embodiments, the case where the present invention is applied to the distributed computing system 100 configured as shown in FIG. 1 has been described. The present invention is not limited to this, and can be widely applied to other distributed computing systems having various configurations.

  In the first to third embodiments described above, the arrangement pattern detection unit that detects various arrangement patterns that distribute and arrange Map processing and Reduce processing in a plurality of virtual machines, and the cost for each arrangement pattern are calculated. 3 of a cost calculation unit and a distributed arrangement unit that selects an arrangement pattern that minimizes the cost based on the calculation result of the cost calculation unit and distributes map processing and reduce processing to a plurality of virtual machines according to the arrangement pattern. The case where one function is realized by one job placement processing unit 114 has been described. However, the present invention is not limited to this. For example, a plurality of distributed calculation control nodes 101 are prepared, and these three functions are distributed to the plurality of distributed functions. You may make it distribute to the calculation control node 101. FIG.

  The present invention can be applied to distributed computing systems having various configurations that perform distributed computing processing using a system virtualization function.

It is a block diagram which shows the whole structure of the distributed calculation system by 1st-3rd embodiment. It is a block diagram which shows the structural example of the conventional distributed calculation system. It is a flowchart which shows the example of a definition of distributed processing. (A) And (B) is a flow graph which shows the example of distribution arrangement | positioning of the calculation process in the distributed calculation system of FIG. 3 is a flow graph showing an example of a distributed arrangement of calculation processing in the distributed calculation system of FIG. 2. It is a block diagram which shows the structural example of a distributed processing system. It is a flowchart which shows the process sequence of a job arrangement | positioning process. It is a chart used for description of the arrangement pattern set. It is a flowchart which shows the process sequence of a cost calculation process. It is a conceptual diagram which shows the structural example of a condition table. It is a flowchart which shows the process sequence of the element cost calculation process on condition of execution time. It is a flowchart which shows the process sequence of a communication path setting process. It is a flowchart which shows the process sequence of a job summary process.

DESCRIPTION OF SYMBOLS 100 ... Distributed computing system, 101 ... Distributed computing control node, 102, 103 ... Distributed computing node, 104 ... Network, 110, 120, 130 ... Hardware, 110A, 120A, 130A ... CPU, 112 ... ... distributed calculation control system, 113 ... distributed calculation definition, 114 ... job is one processing unit, 120B, 130B ... shared memory, 111, 121, 131 ... external storage device, 122, 132 ... system virtualization function , 123, 124, 133, 134... Virtual machine, 301, 402... Map processing, 302, 403.

Claims (8)

In a distributed calculation control apparatus that distributes and arranges a plurality of second calculation processes constituting the first calculation process in a plurality of virtual machines operating on one or a plurality of physical machines
An arrangement pattern detection unit for detecting various arrangement patterns for distributing and arranging the plurality of second calculation processes in the plurality of virtual machines;
A cost calculation unit for calculating a cost for each arrangement pattern;
A distribution arrangement unit that selects an arrangement pattern that minimizes the cost based on a calculation result of the cost calculation unit, and distributes and arranges the plurality of second calculation processes in the plurality of virtual machines according to the arrangement pattern. A distributed calculation control device. The cost calculation unit
The distributed calculation control apparatus according to claim 1, wherein the cost for each of the arrangement patterns is calculated on the condition of at least one of the execution time, power consumption, and I / O amount. The physical machine is
A shared memory shared by the plurality of virtual machines operating on the physical machine,
The distributed arrangement unit is
Based on the calculation result of the cost calculation unit, a plurality of the second calculation processes having an input / output relationship are arranged in the virtual machine operating on the same physical machine, and a plurality of the input / output relationships are provided. 2. The distributed calculation control device according to claim 1, wherein the corresponding physical machine is set to perform data transfer between the virtual machines in which the second calculation processing is arranged via the shared memory. 3. . The distributed arrangement unit is
The plurality of second calculation processes having an input / output relationship are merged into one calculation process based on a calculation result of the cost calculation unit and arranged in the virtual machine. Distributed calculation control device. In a distributed calculation control method for distributing and arranging a plurality of second calculation processes constituting the first calculation process in a plurality of virtual machines operating on one or a plurality of physical machines,
A first step of detecting various arrangement patterns in which the plurality of second calculation processes are distributed and arranged in the plurality of virtual machines;
A second step of calculating a cost for each arrangement pattern;
A third step of selecting an arrangement pattern that minimizes the cost based on a calculation result of the calculation, and distributing and arranging the plurality of second calculation processes in the plurality of virtual machines according to the arrangement pattern. A distributed calculation control method characterized by the above. In the first step,
The distributed calculation control method according to claim 5, wherein a cost for each of the arrangement patterns is calculated on the condition of at least one of the execution time, power consumption, and I / O amount. The physical machine is
A shared memory shared by the plurality of virtual machines operating on the physical machine,
In the third step,
Based on the calculation result of the cost calculation unit, a plurality of the second calculation processes having an input / output relationship are arranged in the virtual machine operating on the same physical machine, and a plurality of the input / output relationships are 6. The distributed calculation control method according to claim 5, wherein the corresponding physical machine is set so that data transfer between the virtual machines in which the second calculation processing is arranged is performed via the shared memory. . In the third step,
The distributed calculation control method according to claim 5, wherein a plurality of the second calculation processes having an input / output relationship are merged into one calculation process and arranged in the virtual machine.