JP6880774B2 - Communication systems, distributed computing systems, nodes, information sharing methods and programs - Google Patents

Description

The present invention relates to a communication system, a distributed computing system, a node, an information sharing method and a program, and more particularly, a communication system, a distributed computing system, a node, an information sharing method and a program responsible for communication between a plurality of nodes constituting the distributed computing environment. Regarding.

Machine learning has been actively studied in recent years. Machine learning aims to learn regularity from a large number of data and make predictions for unknown events. The prediction formula (also called a model) is represented by parameters, and the values of the parameters are optimized based on the data so that the data can be explained correctly. The number of parameters in the prediction formula ranges from a few to tens of millions at most.

Attention is focused on learning sparse data and models. Specific application destinations include document analysis and product recommendation. Sparse data is a word that has a large number of dimensions that characterize the data (for example, the number of vocabularies contained in a sentence, on the order of tens of millions), but is contained in individual samples of data (individual documents in document analysis). Is characterized by a small number. When learning sparse data, the number of parameters becomes enormous because the parameters of the model correspond to the characteristics of the data.
As the amount of data increases, the processing time increases. Therefore, machine learning may be executed on distributed compute nodes. It reduces the amount of calculation processing in a single node and shortens the time to complete execution. For example, Patent Document 1 discloses a configuration in which machine learning is performed using a distributed computer system.

There are two methods for optimizing model parameters in machine learning: the gradient descent method, which uses the entire data in a single process, and the stochastic gradient descent method, which uses only part of the data. A subset of the data used for processing is called a "mini-batch". The stochastic gradient descent method uses a predetermined mini-batch (changes the mini-batch to be used each time) instead of using the entire data when calculating the updated part of the parameter. When this is done on a distributed node, the calculation results are aggregated and the parameters are updated for each calculation using each mini-batch, and the updated parameters are communicated so that they can be used for the calculation using the next mini-batch. For example, Patent Document 2 discloses a parameter learning device that includes a plurality of calculation node units and a parameter update processing unit and performs parameter learning by a stochastic gradient descent method.

Other Non-Patent Documents 1 and 2 propose a parameter server that communicates and manages parameters when performing machine learning in a distributed node environment. The parameter server operates in cooperation with the distributed calculation nodes and sends parameters by notification of the calculation nodes. The calculation node performs the calculation and sends (the amount of change) of the changed parameter to the parameter server.

Japanese Unexamined Patent Publication No. 2012-22558 Japanese Unexamined Patent Publication No. 2012-79080

Mu Li, et al. "Scaling distributed machine learning with the parameter server." 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). 2014. Mu Li, et al. "Communication efficient distributed machine learning with the parameter server." Advances in Neural Information Processing Systems. 2014.

The following analysis is given by the present invention. Consider applying the stochastic gradient descent method in machine learning for sparse data. The sparse data has a very small number of features in the sample. If the mini-batch does not contain many samples, the number of features contained in the mini-batch will be smaller than the whole, and the parameters updated by the calculation using the mini-batch will also be smaller than the whole.

In Non-Patent Documents 1 and 2, the result of calculation performed by the calculation node is once transmitted to the server, and the value is transmitted to the required calculation node in the next step. In Patent Documents 1 and 2, the distribution model update unit and the parameter update processing unit collect and distribute the parameters, respectively.

However, if there is only one compute node that needs a value in the next step, there is no need to go through the process of sending it to the server and sending it to the compute node, and send it directly to the node that needs the value. Is enough. As described above, in the method of aggregating and distributing the elements represented by the parameter server, unnecessary communication occurs in some parts.

The present invention provides a communication system, a distributed calculation system, a node, an information sharing method and a program that can contribute to a reduction in the amount of communication for exchanging intermediate calculation results such as parameters when calculating parameters in a distributed calculation environment. With the goal.

According to the first viewpoint, the element calculated by one node constituting the distributed calculation environment in which the element is calculated using the data in charge of each of the plurality of nodes is directly directed to the node that requires the element next. A communication system including a direct communication determination means for determining whether or not communication is possible is provided. The communication system further includes a direct communication planning means that plans direct communication of elements between the nodes (direct communication plan) based on the result of the determination of the direct communication determination means. Further, the communication system includes a communication means for realizing communication between the nodes according to the direct communication plan.

According to the second viewpoint, a distributed calculation system including the above-mentioned communication system and one node constituting a distributed calculation environment for calculating elements using data in charge of each of the plurality of nodes is provided.

According to the third viewpoint, a node having a function corresponding to the above-mentioned communication system is provided.

According to the fourth viewpoint, a computer placed in a distributed calculation environment that calculates elements using data in charge of each of a plurality of nodes, and an element calculated by one node constituting the distributed calculation environment are next. Based on the step of determining whether or not the element can be directly communicated with the node requiring the element and the result of the determination, the direct communication of the element between the nodes is planned (direct communication plan). A method of sharing information between nodes is provided, including a step and a step of realizing communication between the nodes according to the direct communication plan. This method is tied to a specific machine, a computer that is placed in a distributed computing environment and controls communication to elements.

According to the fifth viewpoint, the elements calculated by one node constituting the distributed calculation environment are next to the computer arranged in the distributed calculation environment in which the elements are calculated using the data in charge of each of the plurality of nodes. Based on the process of determining whether or not the element can be directly communicated with the node requiring the element and the result of the determination, the direct communication of the element between the nodes is planned (direct communication plan). A program for executing a process and a process for realizing communication between the nodes according to the direct communication plan is provided. The program can be recorded on a computer-readable (non-transient) storage medium. That is, the present invention can also be embodied as a computer program product.

According to the present invention, it is possible to reduce the amount of communication for exchanging intermediate calculation results such as parameters when calculating parameters in a distributed calculation environment.

It is a figure which shows the structure of one Embodiment of this invention. It is a figure which shows the structural example of the distributed calculation system to which this invention is applied preferably. It is a figure which shows the structure of the communication system of 1st Embodiment of this invention. It is a figure which shows an example of the information (update target element table) held in the update element storage part of the communication system of 1st Embodiment of this invention. It is a figure which shows an example of the information (reduce communication element table) held in the communication information recording part of the communication system of 1st Embodiment of this invention. It is a figure which shows an example of the information (Broadcast communication element table) held in the communication information recording part of the communication system of 1st Embodiment of this invention. It is a figure which shows an example of the information (direct communication element table) held in the communication information recording part of the communication system of 1st Embodiment of this invention. It is a figure which shows an example of the information (in charge range table) held in the communication information recording part of the communication system of the 1st Embodiment of this invention. It is a flowchart which showed the operation of the communication system of 1st Embodiment of this invention. It is a figure which shows the structure of the communication system of the 2nd Embodiment of this invention. It is a figure which shows the detailed structure of the direct communication determination means of the communication system of the 2nd Embodiment of this invention. It is a figure which shows the detailed structure of the direct communication planning means of the communication system of the 2nd Embodiment of this invention. It is a flow chart which showed the operation of the communication system of the 2nd Embodiment of this invention. It is a flowchart which showed the detail of step S201 of FIG. It is a flowchart which showed the detail of step S203 of FIG. It is another flowchart which showed the detail of step S203 of FIG. It is a flowchart which showed the detail of step S204 of FIG. It is another flowchart which showed the detail of step S204 of FIG. It is a flowchart which showed the detail of step S205 of FIG. It is another flowchart showing the details of step S205 of FIG.

First, an outline of one embodiment of the present invention will be described with reference to the drawings. It should be noted that the drawing reference reference numerals added to this outline are added to each element for convenience as an example for assisting understanding, and the present invention is not intended to be limited to the illustrated embodiment. Further, the connecting line between blocks such as drawings referred to in the following description includes both bidirectional and unidirectional. The one-way arrow schematically shows the flow of the main signal (data), and does not exclude interactivity.

In one embodiment of the present invention, as shown in FIG. 1, a communication system including a direct communication determination means 11, a direct communication planning means 12, and a communication means 13 can be adopted. More specifically, in the direct communication determination means 11, the element calculated by one node constituting the distributed calculation environment for calculating the element using the data in charge of each of the plurality of nodes (calculation node 2 in FIG. 1) is used. Then, it is determined whether or not the element can be directly communicated with the required node. Whether or not the direct communication is possible can be determined by whether or not it is sufficient to directly transmit the calculated element to the next calculation node.

The direct communication planning means 12 plans the direct communication of the elements between the nodes based on the result of the determination of the direct communication determining means. The direct communication plan can be described by a combination of a source compute node, a destination compute node, an element to be transmitted, and a computational step in which the communication is performed. Also, in this direct communication plan, the plan is modified so that the element is not transmitted to other compute nodes at the relevant step.

The communication means 13 realizes communication between the nodes according to the direct communication plan. This communication between the nodes can typically be achieved by the communication system instructing the compute nodes to directly transmit the elements.

According to the communication system 1 as described above, the element determined to be directly communicable is directly transmitted to the next calculation node. As a result, it is possible to reduce the amount of communication required for aggregating the elements and distributing them among the computing nodes.

[First Embodiment]
Subsequently, the first embodiment of the present invention will be described in detail with reference to the drawings. First, a distributed calculation system in which the communication system of the present invention is used will be described. FIG. 2 is a diagram showing a configuration example of a distributed calculation system. With reference to FIG. 2, a distributed calculation system composed of a plurality of calculation nodes 2 and a network connecting the calculation nodes 2 is shown. As the calculation node 2, a general computer composed of a CPU (Central Processing Unit), a memory, an auxiliary storage device, an IO (Input / Output) bus, and the like can be used.

Each compute node 2 holds an optimization element. This optimization element corresponds to the parameters of the prediction model, taking machine learning as an example. The calculation node 2 repeats the calculation so as to set the optimization element to the optimum value. The calculation node 2 obtains an update difference for optimizing the optimization element each time the calculation is performed, and adds it to the optimization element.

At each calculation iteration (step), the calculation node 2 shares the calculated update difference with other calculation nodes. The network is used to share the calculated update difference.

An example of a method of sharing the update difference by the plurality of calculation nodes 2 will be described. In the calculation node 2, a part of the optimizing element is a part in charge of aggregation (hereinafter, the part in charge of aggregation is also simply referred to as a part in charge). The part in charge is generally specified by a range (in that case, the part in charge is called the range in charge). The parts in charge of each compute node are unique and complete. The calculation node performs processing on the optimizing element and shares the calculation result through the respective responsible parts.

The communication system of the first embodiment of the present invention is responsible for sharing the calculation results performed by the distributed calculation system. The communication system stores the elements that each node of the distributed computing system updates at each step (see FIG. 4). It also stores the part in charge of each calculation node (see FIG. 8).

The communication system transmits the difference of the elements updated by each calculation node 2 to the calculation node 2 having the aggregation charge portion including the elements each time the calculation of each step is completed. This is called reduce communication. After that, the compute node adds the elements sent from other compute nodes to the element in charge (aggregation). The calculation node 2 transmits each of the added elements to the calculation node required in the next step, that is, the calculation node 2 that updates the element in the next step. This is called broadcast communication. The following description will be given on the assumption that all the above communication operations are blocking communication.

The communication system described above can typically be realized as a parameter server that sends and receives parameters between the calculation nodes 2. Further, the above-mentioned communication system does not have to be physically independent of the computing node, and may be implemented as a separate program on the same computer as the computing node, for example.

FIG. 3 is a diagram showing a configuration of a communication system according to the first embodiment of the present invention. With reference to FIG. 3, a configuration including a direct communication determination means 110, a direct communication planning means 120, an update element storage unit 130, a communication information recording unit 140, and a communication means 150 is shown.

First, the structure of the data held in the update element storage unit 130 and the communication information recording unit 140 will be described.

As shown in FIG. 4, the update element storage unit 130 holds a table for the number of steps, with the step for which the distributed calculation system performs the calculation as an ID. In each table, the element to be updated (update target element) in each calculation node is recorded in a specific step as an ID. In the example of FIG. 4, the calculation node 2 is designated by the node number. The element to be updated is also specified by an integer (index). For example, in the example of FIG. 4, in step m, it is shown that the compute node with node number = 1 updates the element specified by the numbers 45, 289, and 358.

The communication information recording unit 140 holds a reduce communication element table, a broadcast communication element table, a direct communication element table, and a range of responsibility table. These are referred to when the communication means 150 communicates. Each table will be described below.

In the reduce communication element table, as shown in FIG. 5, the elements that are redue-communicated from the transmitting node to the receiving node are described in each step. The reduce communication element table is prepared for the number of steps calculated by the calculation node 2, and is referred to at the time of reduce communication. The element to be communicated is specified by an index in the same manner as the element to be updated. As the element received by the receiving node, the element included in the range in charge of the receiving node is set. As a result, each receiving node adds the difference of the received element to the element of the part in charge and aggregates the elements of its own range.

As shown in FIG. 6, in the broadcast communication element table, elements that are broadcast from the transmitting node to the receiving node are described in each step. The broadcast communication element table is prepared for the number of steps calculated by the calculation node 2, and is referred to during broadcast communication. The element is specified by an index as in the previous reduce communication element table. In the broadcast communication element table, the element transmitted by the transmitting node is set to the element included in the range in charge of the transmitting node. As a result, the element of the range in charge of each transmitting node is transmitted to the computing node that requires the element in the next step.

In the following description, the read communication element table and the broadcast communication element table do not consider whether or not there is one node that updates a certain element in a certain step, and the elements are aggregated in the necessary calculation nodes. And be described as being set to be distributed.

As shown in FIG. 7, the direct communication element table stores the elements that are directly communicated from the transmitting node to the receiving node in each step. The direct communication element table is also prepared for the number of steps calculated by the calculation node 2 and is referred to during direct communication. Here, the direct communication is a communication method for directly communicating from the updated node to the next required node for a certain element without going through the processes of reade communication and broadcast communication. .. Prior to the start of operation of the communication system, the communication index fields of the direct communication element table are all null.

FIG. 8 is a diagram showing an example of a range table in charge. In the example of FIG. 8, the range of responsibility of each node is described by the range of responsibility start index. In this case, the responsible range of a node is from the responsible range start index to the responsible range start index of the node with the next number.

Subsequently, the operations of the update element storage unit 130, the direct communication determination means 110, the direct communication planning means 120, and the communication means 150 using the information stored in the communication information recording unit 140 will be described.

The direct communication determining means 110 determines in a calculation step performed at each node constituting the distributed calculation system whether or not a predetermined element can be changed to direct communication regardless of readuse communication or broadcast communication. In the present embodiment, when there is only one calculation node that updates the element x in the step m, it is determined in the step m that the element x can communicate directly.

The direct communication planning means 120 changes the contents of the communication element table with respect to the elements determined by the direct communication determination means 110 to be capable of direct communication. Assuming that the element determined to be capable of direct communication in step m is x, the direct communication planning means 120 deletes the latest read communication and broadcast communication related to the element x. Deleting the communication means deleting the corresponding element index from the communication index fields of the reduse communication element table and the broadcast communication element table so as not to perform the communication. Then, the direct communication planning means 120 adds the direct communication of the element x in the latest step in which the element x is updated. Here, adding communication means adding the index of the corresponding element to the communication index field of the direct communication element table so as to perform communication.

The communication means 150 controls the communication of elements between the calculation nodes 2 based on the contents of the communication element table stored in the communication information recording unit 140.

Each part (processing means) of the communication system shown in FIGS. 1 and 3 causes a computer constituting the communication system (which may be a calculation node) to execute each of the above-mentioned processes by using the hardware thereof. It can also be realized by a computer program.

Subsequently, the operation of the present embodiment will be described in detail with reference to the drawings. FIG. 9 is a flowchart showing the operation of the communication system when receiving an instruction to create a direct communication plan. Referring to FIG. 9, first, the direct communication determination means 110 of the communication system determines whether or not there is only one calculation node that updates the element x in step m (steps S101 to S102). For example, if the table of step m of the update element storage unit 130 is scanned and there is only one calculation node that updates the element x, it is determined that the element x can be changed to direct communication (determination Yes in step S102). ). On the contrary, when there are 0 or 2 or more calculation nodes for updating the element x, it is determined that the direct communication cannot be changed (determination No. in step S102).

In the case of the determination Yes in step S102, the direct communication determination means 110 notifies the direct communication planning means 120 of the information of the element x of the step m determined to be changeable to the direct communication. The direct communication planning means 120 deletes the corresponding element index from the communication index fields of the reduce communication element table and the broadcast communication element table (step S103). For example, by deleting the element x (for example, the element of # 4012 in FIG. 5) from the reduce communication element table of FIG. 5, the element is excluded from the target of the reduce communication. Similarly, by deleting the element x from the broadcast communication element table of FIG. 6, the element is excluded from the target of broadcast communication.

Next, the direct communication planning means 120 adds the corresponding element index to the direct communication element table in step m (step S104). The place to be added to the direct communication element table is that the transmitting node of the deleted entry in the reduse communication element table or the broadcast communication element table is the transmitting node, and the receiving node of the deleted entry in the reduse communication element table or the broadcast communication element table is the receiving node. It is the position to be. For example, when it is determined that the element # 245 updated in step m1 can be directly communicated, the element # 245 is deleted from the redue communication element table in step m1 in FIG. 5, and the direct communication element table in step m1 in FIG. 7 is transmitted. The process of adding # 245 to the fields of node = 2 and receiving node = 1 will be performed.

The direct communication determining means 110 and the direct communication planning means 120 carry out the above processing for all the elements of all the steps. That is, the determination is repeated by starting from step m = 0, increasing the element x by one from 0, and when the determination for the element of a certain step is completed, the step m is increased by one and all the elements x are again. Determine if direct communication is possible. Finally, when the determination is completed for all the elements of all the steps, the loop of FIG. 9 is exited and the process of creating the direct communication plan is completed.

As described above, according to the present embodiment, in the distributed calculation system, it is possible to extract all the wasteful communication and switch to the direct communication.

[Second Embodiment]
Subsequently, a second embodiment in which the concept of the neighborhood update step can be introduced to create a more efficient direct communication plan will be described in detail with reference to the drawings. Hereinafter, the differences from the first embodiment will be mainly described.

FIG. 10 is a diagram showing a configuration of a communication system according to a second embodiment of the present invention. The difference from the communication system of the first embodiment shown in FIG. 3 is that the functions of the direct communication determination means 210 and the direct communication planning means 220 have been changed. Hereinafter, the two components will be described in detail.

FIG. 11 is a diagram showing a detailed configuration of the direct communication determination means 210 of the present embodiment. As shown in FIG. 11, the direct communication determination means 210 includes an update node counting means 211 and a single node update determination means 212.

The update node counting means 211 counts the number of updates of a predetermined element in a predetermined step. Specifically, when the update node counting means 211 inputs the node update count step and the element index, the update node counting means 211 displays a table corresponding to the specified number of steps from the update element storage unit 130. With reference, it returns the number of compute nodes for which the element is updated in the step.

The single node update determination means 212 determines whether or not a predetermined element has one node calculated in a predetermined step. Specifically, the single node update determination means 212 takes the output of the update node counting means 211 as an input, performs determination, and outputs a calculation node determined to have one node.

FIG. 12 is a diagram showing a detailed configuration of the direct communication planning means 220 of the present embodiment. As shown in FIG. 12, the direct communication planning means 220 includes a neighborhood update step specifying means 221, a reduse communication deleting means 222, a broadcast communication deleting means 223, and a direct communication adding means 224.

The direct communication planning means 220 of the present embodiment has a forward operation and a backward operation, and when the operations are performed, both processes are performed. In the forward direction, the communication information before the step of the element determined to be directly communicable is modified. In the case of the backward direction, the communication information after the step of the element determined to be directly communicable is corrected.

The neighborhood update step specifying means 221 identifies the step closest to the step m among the steps in which the element x has been updated from the elements x and the step m determined to be capable of direct communication. More specifically, an element, a step, and a direction (forward or backward) are input to the neighborhood update step specifying means 221. When the forward direction is specified, the neighborhood update step identifying means 221 updates the specified element before the specified step, the step closest to step m, and when the backward direction is specified, the specified step. After that, the step closest to step m, in which the specified element is updated, is obtained.

The function of the neighborhood update step specifying means 221 will be described in more detail in the case of the forward direction. It is assumed that the specified step is m and the element is x. The neighborhood update step specifying means 221 confirms whether or not the element x has been updated in order from step m in the forward direction (the step being confirmed is referred to as a "confirmation step"). In the confirmation step, when the element x is not updated, the neighborhood update step specifying means 221 confirms the step immediately before the confirmation step. In the confirmation step, when the element x is updated at any node, the neighborhood update step specifying means 221 outputs the confirmation step as a neighborhood update step (hereinafter, also referred to as step m0).

The reduce communication deleting means 222 deletes the reduce communication that can be replaced with the direct communication for the element determined that the direct communication is possible. The reduce communication deleting means 222 includes an element x determined to be capable of direct communication, a step m, a calculation node a whose element x is updated in step m, and a neighborhood update step specified by the neighborhood update step specifying means. m0 is input. The reduce communication deleting means 222 identifies the node whose element x has been updated in step m0 when step m0 is in the forward direction of m (this is equivalent to m0 <m) (referred to as A0). The reduce communication deleting means 222 deletes the reduce communication of the element x whose source is the calculation nodes belonging to the set A0 in step m0.

On the other hand, when step m0 is in the backward direction of m (which is equivalent to m0> m), the reduce communication deleting means 222 deletes the reduce communication of the element x whose source is the calculation node a in step m. In the above description, deleting the reduce communication of a predetermined element means that the corresponding element index is selected from the communication index field of the reduce communication element table of the corresponding step of the communication information recording unit 140 so as not to perform the communication. It means to delete.

The broadcast communication deleting means 223 deletes the broadcast communication that can be replaced with the direct communication for the element determined that the direct communication is possible. The broadcast communication deleting means 223 includes an element x determined to be capable of direct communication, a step m, a calculation node a whose element x is updated in step m, and a step m0 which is an output of the neighborhood update step specifying means 221. Entered. The broadcast communication deleting means 223 transmits a calculation node (referred to as b) including the element x in the responsible portion in the step (m-1) when the step m0 is in the forward direction of m (this is equivalent to m0 <m). The original broadcast communication of element x is deleted.

When step m0 is in the backward direction of m (which is equivalent to m0> m), the broadcast communication deleting means 223 deletes the broadcast communication of the element x whose source is the calculation node b in step (m0-1). .. In the above description, deleting the broadcast communication of a predetermined element means that the corresponding element index is selected from the communication index field of the broadcast communication element table of the corresponding step of the communication information recording unit 140 so as not to perform the communication. It means to delete.

The direct communication adding means 224 adds the direct communication to the element determined to be capable of the direct communication. The element x determined to be capable of direct communication, the calculation node a whose element x is updated in step m, and the step m0 which is the output of the neighborhood update step specifying means are input to the direct communication adding means 224. Will be done. The direct communication addition means 224 specifies a calculation node (generally a plurality) for updating the element x in step m0 when step m0 is in the forward direction of m (which is equivalent to m0 <m). In step m0, the direct communication adding means 224 adds the direct communication of the element x with the calculation node as the source and the calculation node a as the transmission destination.

On the other hand, when step m0 is in the backward direction of m (which is equivalent to m0> m), the direct communication addition means 224 specifies a calculation node (generally a plurality) for updating element x in step m0. Then, in step m, the direct communication adding means 224 adds the direct communication of the element x with the calculation node a as the source and the calculation node as the transmission destination.

Subsequently, the operation of the communication system of the present embodiment will be described in detail with reference to the drawings. FIG. 13 is a flowchart showing the operation of the communication system according to the second embodiment of the present invention. The basic operation is the same as that of the first embodiment, that is, the operation of the direct communication determination means 210 (step S201), the operation of the neighborhood update step specifying means of the direct communication planning means 220 (step S203), and the reduce communication deletion means. The operation of the broadcast communication deleting means (step S204) and the operation of the direct communication adding means (step S205) are performed. Similar to the first embodiment, the communication system repeats the process m × x times with the number of steps being m and the element being x, and performs the process of switching the element determined to be directly communicable to direct communication. Hereinafter, the operation in one loop will be described in detail.

FIG. 14 is a flowchart showing the operation of the direct communication determination means 210. Referring to FIG. 14, the update node counting means 211 counts the compute nodes whose element x is updated in step m (step S2101). The single node update determination means 212 determines whether the number of updated calculation nodes is 1 or not (step S2102). When it is determined that the updated calculation node is 1, the single node update determination means 212 outputs the updated calculation node.

The following is an explanation in the case where Yes is set in step S202 of FIG. In the following description, let a be the calculation node output by the direct communication determination means 210. 15 and 16 are flowcharts showing the operation of the neighborhood update step specifying means 221 of the direct communication planning means 220, FIG. 15 shows the neighborhood update step search process in the front direction, and FIG. 16 shows the neighborhood update step search process in the rear direction. The neighborhood update step search process is shown.

First, with reference to FIG. 15, the search process in the forward direction by the neighborhood update step specifying means 221 will be described. First, the neighborhood update step specifying means 221 substitutes m-1 for the confirmation step m0 (step S2311). Next, the neighborhood update step specifying means 221 determines whether or not there is a calculation node whose element x has been updated in step m0 (step S2312). As a result of the determination, when there is a calculation node whose element x has been updated in step m0 (Yes in step S2312), the neighborhood update step specifying means 221 outputs m0 and ends the operation (step S2313).

On the other hand, when there is no calculation node whose element x has been updated in step m0 (No in step S2312), the neighborhood update step identifying means 221 determines whether step m0 is the first element, that is, step m0 is the first step. Determine (step S2314). When m0 is the first step (Yes in step S2314), there is no step in which the element x is updated in the forward direction. Therefore, the neighborhood update step specifying means 221 includes the direct communication planning means 220 including the subsequent processing. Ends the search operation in the forward direction of (step S2315). On the other hand, when step m0 is not the first step (No in step S2314), the neighborhood update step specifying means 221 reduces the value of m0 by one (step S2316). Then, the process returns to the determination in step S2312. As described above, the neighborhood update step specifying means 221 searches for step m0 in which the element x is updated before the step m in which the element x is updated. The above is the search operation in the forward direction of the neighborhood update step specifying means 221.

The operation of the neighborhood update step specifying means 221 in the backward direction is substantially the same as the processing in the forward direction. Specifically, the neighborhood update step specifying means 221 increases m0 one by one until the step m0 that updates the element x is found, and the step m0 in which the element x is updated after the step m in which the element x is updated is found. I will search for. If m0 becomes the final step in the process, the search operation in the backward direction including the subsequent processing is terminated (step S2325). The above is the backward search operation of the neighborhood update step specifying means 221.

Subsequently, the operations of the read communication deleting means 222 and the broadcast communication deleting means 223 in step S204 of FIG. 13 will be described. 17 and 18 are flowcharts showing the operations of the reduse communication deleting means 222 and the broadcast communication deleting means 223 of the direct communication planning means 220, and FIG. 17 is a communication when the neighborhood update step m0 in the forward direction is found. The deletion process is shown, and FIG. 18 shows the communication deletion process when the neighborhood update step m0 in the backward direction is found.

First, with reference to FIG. 17, the forward communication deletion process by the read communication deletion means 222 and the broadcast communication deletion means 223 will be described. First, the read communication deleting means 222 and the broadcast communication deleting means 223 identify the calculation nodes (generally a plurality of) whose elements x have been updated in step m0 (step S2411). The reduce communication deleting means 222 deletes the reduce communication of the element x whose source is the node in step m0 (step S2412). Next, the broadcast communication deleting means 223 deletes the broadcast communication of the element x in step m-1 (step 2413).

Subsequently, with reference to FIG. 18, the backward communication deletion process by the read communication deletion means 222 and the broadcast communication deletion means 223 will be described. First, the reduce communication deleting means 222 deletes the reduce communication of the element x whose source is the calculation node a in step m (step S2421). Next, the broadcast communication deleting means 223 deletes the broadcast communication of the element x in step m0-1 (step S2422).

Subsequently, the operation of the direct communication adding means 224 in step S205 of FIG. 13 will be described. 19 and 20 are flowcharts showing the operation of the direct communication addition means 224 of the direct communication planning means 220, and FIG. 19 shows a communication addition process when the neighborhood update step m0 in the forward direction is found. Reference numeral 20 denotes a communication addition process when the neighborhood update step m0 in the backward direction is found.

First, with reference to FIG. 19, the forward communication addition process by the direct communication addition means 224 will be described. First, the direct communication addition means 224 specifies a calculation node for updating the element x in step m0 (step S2511). Next, in step m0, the direct communication adding means 224 adds the direct communication of the element x whose source is the previously specified node and whose destination is the calculation node a (step S2512).

Subsequently, with reference to FIG. 20, the backward communication addition process by the direct communication addition means 224 will be described. First, the direct communication adding means 224 specifies the calculation node that updates the element x in step m0 (step S2521). Next, in step m0, the direct communication adding means 224 adds the direct communication of the element x having the previously specified calculation node as the source and the calculation node a as the transmission destination (step S2522).

As described above, according to the present embodiment, in the distributed calculation system, not only the step m and the element x capable of direct communication but also the steps before and after updating the element x are searched for, and the section is also directly communicated. It is possible to switch. Therefore, it is possible to further improve the efficiency of communication as compared with the first embodiment.

Although each embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and further modifications, substitutions, and adjustments are made without departing from the basic technical idea of the present invention. Can be added. For example, the network configuration, the configuration of each element, and the expression form of the message shown in each drawing are examples for assisting the understanding of the present invention, and are not limited to the configurations shown in these drawings.

Finally, a preferred embodiment of the present invention is summarized.
[First form]
(Refer to the communication system from the first viewpoint above)
[Second form]
The communication system described above further includes a communication information recording unit that holds a communication plan between the nodes created in advance and a direct communication plan created by the direct communication planning means, and the communication means is the communication. Based on the communication plan and the direct communication plan held in the information recording unit, it is possible to adopt a configuration that realizes communication between the nodes.
[Third form]
The above-mentioned communication system
Further, it is provided with an update element storage unit that stores the elements to be updated in each node.
The direct communication determination means can adopt a configuration in which it is determined that direct communication is possible when there is one node that calculates an element in a certain calculation step with reference to the update element storage unit.
[Fourth form]
The direct communication planning means of the above-mentioned communication system is
The direct communication determination means identifies a step in the vicinity of the step in which the element has been updated for a set of the step and the element determined to be directly communicable, and from the communication plan, the step in the vicinity and the element are described. It is possible to adopt a configuration in which the indirect communication related to the element with the step determined to be directly communicable is deleted and the direct communication related to the element is added to the communication plan.
[Fifth form]
The communication information recording unit of the communication system described above has a reduce communication element table that records information related to communication that aggregates the calculated elements into a predetermined node, and a node that needs the aggregated elements in the next step. A broadcast element table for spreading broadcast elements and a direct communication element table for recording information relating to direct communication of elements between nodes are provided, and the communication means is provided between the nodes according to the information recorded in the table. It is possible to adopt a configuration that realizes communication.
[Sixth form]
The element of the communication system described above can be a difference value for modifying the parameters of the model used for machine learning.
[7th form]
(Refer to the distributed calculation system from the second viewpoint above)
[8th form]
(Refer to the node from the third viewpoint above)
[9th form]
(Refer to the information sharing method between nodes from the fourth viewpoint above)
[10th form]
(Refer to the computer program from the fifth viewpoint above)
The seventh to tenth forms can be developed into the second to sixth forms in the same manner as the first form.

The disclosures of the above patented documents and non-patented documents shall be incorporated into this document by citation. Within the framework of the entire disclosure (including the scope of claims) of the present invention, it is possible to change or adjust the embodiments or examples based on the basic technical idea thereof. Further, within the framework of the disclosure of the present invention, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or embodiment, each element of each drawing, etc.) are possible. Is. That is, it goes without saying that the present invention includes all disclosure including claims, and various modifications and modifications that can be made by those skilled in the art in accordance with the technical idea. In particular, with respect to the numerical range described in this document, it should be interpreted that any numerical value or small range included in the range is specifically described even if there is no other description.

1,100,200 Communication system 2 Calculation node 11, 110, 210 Direct communication determination means 12, 120, 220 Direct communication planning means 13,150 Communication means 130 Update element storage unit 140 Communication information recording unit 211 Update node counting means 212 One-node update determination means 221 Neighborhood update step identification means 222 reduce communication deletion means 223 broadcast communication deletion means 224 Direct communication addition means

Claims (7)

A distributed calculation environment that calculates the elements that are the parameters of the prediction model using the data that multiple nodes are in charge of based on the plan created in advance, and calculates the parameters of the prediction model by sharing the results. It is a communication system that composes
Each step in the planning refers to the element to be updated in each node, and depending on whether there is one compute node that updates the element in a step, the element then needs the element. A direct communication determination means for determining whether or not a target for direct communication with a node can be used,
Based on the result of the determination of the direct communication determination means, the direct communication planning means for planning the direct communication of the elements between the nodes, and the direct communication planning means.
A communication means that realizes communication between the nodes according to a plan that defines communication for the element to be directly communicated and other elements.
A communication system characterized by comprising. The direct communication planning means
With respect to the set of the step and the element determined to be directly communicable by the direct communication determining means, a step in the temporal vicinity of the step in which the element is updated is specified.
Communication system according to claim 1 to replace in prior Symbol plan, the engaging Ru communication to the elements of the step and the is determined directly communicable with the steps of the neighborhood, the direct communication according to prior Symbol elements. The plan includes a reduce communication element table that records the source and destination of the communication that aggregates the calculated elements into a predetermined node, and a communication that spreads the aggregated elements to the necessary nodes in the next step . Described by a broadcast communication element table that records the source and destination, and a direct communication element table that records the source and destination of the direct communication of elements between nodes.
The communication system according to claim 1 or 2, wherein the communication means realizes communication between the nodes according to the information recorded in each table. The communication system according to any one of claims 1 to 3 , wherein the element is a difference value for modifying a parameter of a model used for machine learning by a stochastic gradient descent method in a distributed calculation environment. The communication system according to any one of claims 1 to 4.
A distributed calculation system that includes one node that constitutes a distributed calculation environment that calculates elements using the data that multiple nodes are in charge of. Based on a pre-created plan, a computer placed in a distributed calculation environment that calculates elements using the data that multiple nodes are in charge of and shares the results to calculate the parameters of the prediction model.
Each step in the planning refers to the element to be updated in each node, and depending on whether there is one compute node that updates the element in a step, the element then needs the element. Steps to determine whether or not it can be targeted for direct communication with the node to be
Based on the result of the determination, the step of planning the direct communication of the elements between the nodes and
A step of realizing communication between the nodes according to a plan that defines communication for the element to be directly communicated and other elements, and
A method of sharing information between nodes, characterized by including. Based on a pre-created plan, a computer placed in a distributed calculation environment that calculates elements using the data that multiple nodes are in charge of and shares the results to calculate the parameters of the prediction model.
Each step in the planning refers to the element to be updated in each node, and depending on whether there is one compute node that updates the element in a step, the element then needs the element. And the process of determining whether or not it can be targeted for direct communication with the node to be
Based on the result of the determination, the process of planning the direct communication of the elements between the nodes, and
A process for realizing communication between the nodes according to a plan that defines communication for the element to be directly communicated and other elements.
A program that executes.

Images