JP5826390B2 - Transfer method and graph processing system - Google Patents

Description

  The present invention relates to graph processing using a plurality of calculation nodes, and more particularly to transfer of graph information between calculation nodes.

  As an information compression technique in graph processing, there is a technique described in Non-Patent Document 1. The technique described in Non-Patent Document 1 is that a vector A that stores a non-zero value, a vector B that stores a column number of a non-zero location, and the first non-zero location of each row is the vector A The information is compressed by expressing the graph structure data with a vector C storing whether it corresponds.

Richard Barrett, et al., "Templates for linear system solutions: Building for the Solutions of Linear Systems for Iterators," Society for Industrial and Applied Materials (USA), 1994, pp. 64-65

  With the technique described in Non-Patent Document 1, it is possible to compress graph structure data, but it is not possible to compress information that propagates between vertices of a graph, which is intermediate data during graph processing. In the case of graph processing using a plurality of calculation nodes, the inventors of the present application transfer intermediate data because the amount of intermediate data transferred between calculation nodes increases as the scale of the graph to be processed increases. It has been found that there is a problem that the time required for the processing becomes longer and the overall speed of the graph processing becomes slower.

  Accordingly, an object of the present invention is to transfer intermediate data for efficient graph processing.

The present invention relates to a transmission buffer that accumulates intermediate data in each computation node for transfer of intermediate data having a pair of information of a graph vertex of a transmission source and information of a graph vertex of a transmission destination in graph processing by a plurality of computation nodes. And sort the sequence of intermediate data in the group of accumulated intermediate data based on the information that increases the compression rate after sorting either the source graph vertex information or the destination graph vertex information After the sorting , the above-mentioned problem is solved by compressing the accumulated intermediate data group and transferring the intermediate data group.

  According to the present invention, it is possible to realize efficient transfer of intermediate data when performing graph processing with a plurality of computation nodes.

It is a figure which shows the graph processing system which concerns on the Example of this invention. It is a figure which shows the system configuration | structure of the Ap server of FIG. It is a block diagram which shows the internal structure of the server apparatus of FIG. It is a figure which shows the transmission buffer and reception buffer which were prepared on the memory device of FIG. It is a figure which shows the example of the graph of a process target. FIG. 6 is a diagram expressing the graph of FIG. 5 in a matrix format. FIG. 6 is a diagram expressing the graph of FIG. 5 in a CSR format. It is the figure which showed the structure of the transfer information between vertices. It is a figure which shows the example of vertex allocation information. It is a flowchart which shows the example of a graph process. 11 is a flowchart showing details of the compression processing of FIG. 10. FIG. 11 is a flowchart showing details of the decompression process of FIG. 10. FIG. It is explanatory drawing which showed the example of which server each vertex of the graph of FIG. 5 is processed. It is the figure which showed the propagation of the information between vertices with respect to FIG. It is explanatory drawing which shows the example before the sorting of the information transferred from the server apparatus 420 to the server apparatus 430. FIG. It is explanatory drawing which shows the example after the sorting of the information transferred from the server apparatus 420 to the server apparatus 430. FIG. FIG. 11 is an explanatory diagram illustrating an example after compression of information transferred from the server apparatus 420 to the server apparatus 430.

  FIG. 1 is a diagram schematically showing a configuration of a graph processing system 10 according to an embodiment of the present invention. As shown in FIG. 1, the graph processing system 10 includes a client 100 such as a PC or a portable terminal that can communicate via the Internet 200, a web server (Web server) 300 that receives a request from the client 100 via the Internet 200, An application server (Ap server) 400 that performs graph analysis in response to a request from the Web server 300 and a database server (DB server) 500 that accesses a database are included. However, the network connecting the client 100 and the Web server 300 is not limited to the Internet 200, and may be a LAN, for example. In the present embodiment, a web three-layer system including the Web server 300, the Ap server 400, and the DB server 500 is described, but the present invention is not limited to this configuration. For example, a two-layer configuration of the Web server 300 and the DB server 500 that also performs graph analysis may be used.

  FIG. 2 is a diagram showing a system configuration of the Ap server 400 of FIG. As shown in FIG. 2, the Ap server 400 includes server devices 420, 430, 440, and 450 and a network device 410 that connects them. Each of the server apparatuses 420 to 450 serves as a calculation node in the graph processing. Since the number of server devices is determined by the performance of the server device and the task load, the number of server devices is not limited to the configuration of four devices and may be other numbers. The server apparatuses 420 to 450 execute graph processing such as solving the shortest path problem while performing parallel processing and communicating with each other. Server identification information (server ID) is given to each of the server devices 420-450. In this embodiment, three server devices are used for graph processing, the server ID of the server device 420 is “1”, the server ID of the server device 430 is “2”, and the server ID of the server device 440 is “3”. .

  FIG. 3 is a block diagram showing an internal configuration of the server apparatus 420 shown in FIG. The internal configuration of the server apparatuses 430, 440, and 450 only needs to have a function equivalent to that of the internal configuration of the server apparatus 420. For example, the server apparatuses 430, 440, and 450 may have different manufacturers and performance. Hereinafter, the server apparatus 420 will be described as a representative.

  The server device 420 includes a central processing unit (CPU) 600, a memory device 610, a storage device 620, an input device 630, an output device 640, a network interface (I / F) 650, and a bus 660. Data transfer between components in the server apparatus 420 is mainly performed via the bus 660. The CPU 600 uses the memory device 610 to execute a graph analysis program and to control the overall operation of the server device 420. The memory device 610 is a primary storage device such as an SDRAM, and holds instructions and data necessary when the CPU 600 executes a program. The storage device 620 is a secondary storage device such as an HDD or an SSD, and holds programs and data for a long period of time and is also used as a swap area of the memory device 610. The input device 630 is a mouse or a keyboard, and the output device 640 is a display device or a speaker. The network interface 650 is used for communication with other server apparatuses, and InfiniBand can be used.

  FIG. 4 shows a NEXT that stores a transmission buffer 611, a reception buffer 612, a processing queue 613, a graph calculation module 614, vertex assignment information 615, and processing for the next round, which are secured in the memory area of the memory device 610. FIG. 6 shows a queue 616, a compression module 617, and an expansion module 618. The transmission buffer 611 is a memory area for temporarily storing data to be transmitted when communication is performed between server apparatuses. The reception buffer 612 is a memory area for temporarily storing data received when communication is performed between server apparatuses. The intermediate data of the graph processing transferred between the server apparatuses is temporarily stored in the transmission buffer 611 and the reception buffer 612. The data stored in the transmission buffer 611 is transferred to the network interface 650 according to a transmission instruction from the CPU 600 and then transmitted to another server device via the network device 410. The transmission data is received by the network interface 650 of another server device, and then stored in the reception buffer 612 of another server device. The processing queue 613 is a memory area for temporarily storing data that can be calculated in a phase of graph processing, and the NEXT queue 616 is a memory area for temporarily storing data that can be calculated in the next phase. . The graph calculation module is a program module that performs calculation of graph processing such as shortest path search. As will be described later, the compression module 617 is a program module that compresses intermediate data for graph processing transferred between server apparatuses. As will be described later, the decompression module 618 is a program module that decompresses intermediate data for graph processing transferred between server apparatuses. As will be described later, the vertex assignment information 615 is information indicating the assignment of graph processing to a plurality of server devices.

  FIG. 5 is a graph showing a specific example for explaining the operation of the graph processing system 10 of this embodiment. Circles in FIG. 5 represent vertices of the graph, and vertex identification numbers (hereinafter referred to as vertex numbers) are shown in the circles. Lines between vertices are graph edges. Graphs are suitable for expressing relationships between various things. For example, if the vertices of the graph are regarded as stations or intersections, the edges represent tracks and roads, and if the vertices are regarded as people or companies, the edges indicate the interrelationship between people and companies. The edge has a weight indicating the relationship between the vertices, and in the former example, it indicates time and distance, and in the latter example, it indicates the strength of connection.

  6 and 7 represent the graph shown in FIG. 5 as graph structure data, and show the same graph structure as the graph of FIG. 6 represents the graph of FIG. 5 in a matrix format, and FIG. 7 represents the graph of FIG. 5 in a CSR (Compressed Sparse Row) format. The values in the table of FIG. 6 indicate edge weights, meaning that there are no edges at locations where the value is 0 (it may be more convenient to express ∞ depending on the problem to be solved such as the shortest path problem). To do. The CSR format in FIG. 7 stores values storing non-zero values, columns storing column numbers of non-zero locations, and what values of values correspond to the first non-zero location of each row. It consists of rowindex. The CSR format is suitable for expressing a sparse matrix as shown in the figure. In the matrix format, an n × m storage capacity is required to represent an n × m matrix. However, in the CSR format, if the number of non-zeros is 1, a storage capacity of 2 × l + n + 1 is sufficient. For example, a graph having a scale-free property represented by a big graph is a sparse matrix, and therefore generally has a CSR format or a CSC (Compressed Sparse Column) format (in which the rows and columns are replaced with respect to the CSR format). Graph structure data is saved in a compressed storage format.

  FIG. 8 shows information transferred between the vertices. Each row indicates the vertex number (target) of the destination vertex (target vertex), the vertex number (source) of the source vertex (source vertex), and the transmission. It consists of a set of data. Here, as an example, information transferred from the vertex 3 to the vertices 7, 8, and 9 is shown. The relationship between the target and the destination server device, and the source and the source server device are stored in the memory device 610 of each server device as the vertex assignment information 615.

  FIG. 9 is a diagram showing the vertex assignment information 615 of the present embodiment. In the vertex assignment information 615, a pair of a vertex number and a server ID of a server device to which a vertex calculation of the vertex number is assigned is an entry. The vertex assignment information 615 can be generated when assigning vertices to each server device. In the present embodiment, a group of vertices with younger vertex numbers are arranged in ascending order of server IDs, but the configuration of the arrangement of vertices is not limited to this, and the present invention can be used if the vertices are distributed and arranged in each server. Can be applied.

  FIG. 10 is a flowchart showing an example of the graph processing operation performed by the graph processing system 10 of this embodiment. In this embodiment, the graph structure data of FIG. 7 is stored in the storage device 620 as a processing target, and the graph structure data of FIG. 7 is transferred to the memory device 610 of each server device as shown in FIG. A description will be given on the assumption that each server device is assigned a vertex to be calculated by assignment. The vertex assignment calculated by each server device is stored as vertex assignment information 615 in the memory device 610 of each server device. Also, a description will be given assuming that the vertex that is the starting point of the graph analysis is assigned to the server device 420.

  First, the graph calculation module 614 of the server device 420 selects a vertex that is a starting point of graph analysis as a target vertex (steps S100 and S101). Next, the graph calculation module 614 of the server device 420 creates a set of source vertices and data for the selected target vertices and puts them into the processing queue 613 (step S102). Here, since the target vertex is the start point, the source vertex and data are dummy data (for example, source = target, data = Z).

  In step S103, the graph calculation module 614 of each server device determines whether or not the processing queue 613 of each server device is empty. If the operation of each server device is empty, the process proceeds to step S108. Otherwise, the process proceeds to step S104. In step S104, the graph operation module 614 of each server device extracts a set of (target, source, data) information from the processing queue 613 of each server device, and performs an operation on the target vertex.

  Next, in step S105, the graph calculation module 614 of each server device selects the target vertex that was in the processing queue 613 as the next source vertex, and refers to the graph structure data in FIG. A set of source vertices, target vertices read from the graph structure data, and data, which are intermediate data of processing, is generated. When the graph calculation module 614 of each server device processes the target vertex of the set generated in step 105 in step 106, the server device puts the generated set information in the Next queue 616 and the other server processes it Puts the generated set of information into the transmission buffer 611. The transmission buffer 611 is preferably divided for each server device that processes the target vertex.

  In step S107, in order to efficiently transmit the transmission information collectively, each server device determines whether or not the amount of information stored in the transmission buffer 611 exceeds a preset transmission reference size. When the reference size is exceeded, the transmission of each server device proceeds to start transmission in order to start transmission, and when the reference size is not exceeded, the processing returns to step S103. In step 108, each server device determines whether untransmitted information remains in the transmission buffer 611 in a state where the processing queue 613 is empty. If the transmission buffer 611 is not empty, the operation of each server device proceeds to step S200 in order to transmit untransmitted information. If the transmission buffer is empty, there is no untransmitted information, so that the reception starts in step S300. Proceed to

  FIG. 11 shows steps S201 to S205 which are detailed steps of step S200. First, in step S200, the compression module 617 of each server apparatus counts up the number of common values for each column of the target vertex, the source vertex, and the data in the transmission buffer 611 (step S201, step S202). Thereafter, the compression module 617 of each server device sorts the data in the transmission buffer 611 using the column with the highest compression rate as a key when run-length encoding is performed on the assumption that values common to the columns are continuous. (Step S203). At this time, sorting is performed while maintaining the correspondence between the set of target vertices, source vertices, and data. As described above, the sorting of the set arrangement is performed in the group of the set of the target vertex, the source vertex, and the data that are the intermediate data stored in the transmission buffer 611, so that the transmission data can be efficiently compressed. This makes it possible to realize efficient data transfer, and thus speed up the graph processing. Furthermore, by using the column having the highest compression rate among the target vertex, source vertex, and data as a key, it is possible to compress transmission data more efficiently. Note that the number of values common to each column of the target vertex and the source vertex may be counted up, and the column having the higher compression rate of the target vertex or the source vertex may be sorted as a key. For example, in the case of graph processing in which there is no weight on an edge, transmission of data is not necessary.

  After performing the sorting, the compression module 617 of each server device performs run-length encoding on the key column (step S204). By performing the sorting in advance, common numerical values become continuous, and the compression rate in the run-length encoding can be increased. Further, the compression module 617 of each server device adds information indicating the column as the key to the head of the transmission information so that it can be understood which column was used as the key when decoding by the decompression module 618. In step S109 following step S200, each server device refers to the vertex assignment information 615 and transmits the transmission data processed in step S200 to the server device to which the computation of the target vertex is assigned.

  FIG. 12 shows steps S301 to S306 which are detailed steps of step S300. In the decompression processing step S300, the decompression module 618 of each server device first extracts information from the reception buffer 612, and specifies which column should be decoded from the top information (steps S301 and S302). The decompression module 618 performs run-length composition on the identified column, and decompresses the original information (step S303). In step S304, the decompression module 618 determines whether the identified column is a target vertex. If the identified column is not the target vertex, the decompression module 618 performs sorting using the target vertex as a key (step S305). Also in this case, the sorting is performed while maintaining the correspondence between the set of the target vertex, the source vertex, and the data. In this way, the arrangement of the intermediate data in the group of intermediate data that has arrived at the reception buffer 612 is sorted. If the identified column is the target vertex, since the sorting has already been performed before transmission, the sorting is not performed and the process ends (S306).

  In step S <b> 110, each server device extracts information transmitted from another server device from the reception buffer 612 and puts it in the Next queue 616. If there is no information to be received, each server device does nothing in step S110. Although not described here, whether or not there is information to be received is separately communicated between the server apparatuses. Thereafter, in step S111, each server device determines whether or not the Next queue 616 is empty in all the server devices performing the graph processing. Whether or not the Next queue 616 is empty in all the server devices performing the graph processing depends on whether each Server device has its own Next queue 616 in another server device when its Next queue 616 is empty. Each server device can determine by notifying that it has become empty. If the Next queue 616 is not empty, each server device moves information from the Next queue 616 to the processing queue 613 (Step S112), and the operation of each server device returns to Step S103. If the Next queue 616 of all server devices is empty, the processing for all vertices is completed (step S113).

  By the compression / decompression process of steps S200 and S300, the received information is put into the process queue 613 in a state of being sorted using the target vertex as a key. Since the storage order in the processing queue 613 is the calculation order of each server device with respect to the vertices, the processing for the same target vertex is continuously performed by setting the target vertex order. By performing the processing continuously, it is possible to reduce the probability that the intermediate data is replaced from the cache of the CPU 600 to the memory device 610 or the swap from the memory device 610 to the storage device 620 occurs. Hereinafter, the compression / decompression operation will be described using a specific example.

  FIG. 13 is a diagram showing an example of which server device processes each vertex of the graph of FIG. In this figure, vertices 1 to 10 are divided into a group 700, vertices 11 to 15 are divided into a group 710, and vertices 16 to 25 are divided into a group 720. In this embodiment, it is assumed that the group 700 is processed by the server apparatus 420, the group 710 is processed by the server apparatus 430, and the group 720 is processed by the server apparatus 440.

  FIG. 14 is a diagram showing the flow of information between vertices with arrows when graph analysis is performed with vertex 3 as a starting point with respect to FIG. 13. Hereinafter, the flow of processing for the graph analysis starting from the vertex 3 will be described. First, the graph calculation module 614 of the server apparatus 420 selects the vertex 3 as the target vertex, and puts (target, source, data) = (3, 3, Z) (source and data are dummy data) into the processing queue 613. . The graph operation module 614 of the server apparatus 420 extracts (target, source, data) = (3, 3, Z) from the processing queue 613 and executes the operation on the vertex 3. The graph calculation module 614 of the server device 420 refers to the graph structure data with the vertex 3 as the source vertex, and obtains that the next target vertex to be processed is the vertices 7, 8, and 9. Since the vertices 7, 8, and 9 all belong to the group 700, the graph operation module 614 of the server device 420 stores (target, source, data) = (7, 3, c), (8 in its Next queue 616. 3, d), (9, 3, e). Since there is no information to be transmitted / received, the graph calculation module 614 of the server device 420 moves the contents of the Next queue 616 to the processing queue 613, calculates the vertices 7, 8, and 9 and refers to the graph structure data. Depending on the classification of the group to which each vertex belongs, the graph calculation module 614 of the server device 420 moves to the next queue 616 at the vertex 7 (target, source, data) = (1, 7, a), (2, 7, b). , To the transmission buffer 611 (target, source, data) = (11, 7, i), at the vertex 8, to the transmission buffer 611 (target, source, data) = (11, 8, j), at the vertex 9, the Next queue 616 (Target, source, data) = (4, 9, f), and (target, source, data) = (12, 9, l), (11, 9, k) are input to the transmission buffer 611. The compression module 617 of the server apparatus 420 performs transfer by compressing the contents of the transmission buffer 611 because the processing queue 613 has become empty.

  FIG. 15A shows the state before sorting the contents of the transmission buffer 611, FIG. 15B shows the state after sorting the contents of the transmission buffer 611, and FIG. 15C shows the contents of the transmission buffer 611 sorted and compressed. Each subsequent state is shown. Prior to the start of transmission, the compression module 617 of the server apparatus 420 selects the one with the highest compression rate when each column of (target, source, data) is sorted and run-length encoding is performed. Here, three targets are common to the vertex 11, two are common to the vertex 9, data is not common, and compression of the target in the row direction is selected. Thereafter, the compression module 617 of the server apparatus 420 performs sorting in the row direction (FIG. 15B), and performs run-length encoding on the target (FIG. 15C). In FIG. 15C, the fact that the three vertices 11 are common is expressed as 11 × 3, but this can be expressed as, for example, the number in which the most significant bit is 1 as the number of repetitions. When the target, source, data, and number of repetitions are each represented by a 4-byte variable, the size before compression is 48 bytes, and the size after compression is 44 bytes. Further, 1 byte of information indicating that the target is compressed is added to the head of the transfer information, and 45 bytes of information is transmitted to the server device 430. In the server device 430, the received information is decompressed, and the decompressed data is placed in the Next queue 616. In the server apparatuses 420 and 430, information (target, source, data) is moved from the Next queue 616 to the processing queue 613, and the processing is also advanced for another vertex. Thereafter, since the above-described processing is repeated for each vertex, description thereof is omitted.

  As described above, the calculation for each vertex is performed in the order of the target vertices extracted from the processing queue 613. However, when the calculation for the same target vertex is not performed continuously, the intermediate data is obtained by the calculation for other target vertices. However, if the cache is replaced with a memory device or a swap from the memory device to the storage device occurs, the processing performance deteriorates. Therefore, when compression is performed by source or data, a flow is performed in which sorting is performed by target after decompression at the receiving side.

  As described above, in the transfer method of the present embodiment, it is possible to efficiently compress the transmission data by sorting the arrangement of the intermediate data in the group of intermediate data transferred between the graph vertices. Efficient data transfer can be realized, and the graph processing can be speeded up. Further, by comparing the compression rates when the columns are sorted and compressed in the row direction, the amount of information transferred between servers can be more efficiently reduced. Also, if the source or data is sorted using the source or data as a key and transferred after compression, the amount of intermediate data to be saved in the memory device or storage device can be reduced by re-sorting with the target after decompression on the receiving side. This makes it possible to perform graph analysis at higher speed.

  100: client, 200: Internet, 300: Web server, 400: Ap server, 500: DB server, 410: network device, 420 to 450: server device, 600: CPU, 610: memory device, 611: transmission buffer, 612 : Reception buffer, 613: processing queue, 614: graph operation module, 615: vertex assignment information, 616: NEXT queue, 617: compression module, 618: decompression module, 620: storage device, 630: input device, 640: output device 650: Network interface 660: Bus

Claims (9)

A method for transferring intermediate data in graph processing by a plurality of computation nodes,
Each compute node has a send buffer,
Each compute node is assigned a graph vertex to be processed,
The intermediate data includes a pair of information on the graph vertex of the transmission source and information on the graph vertex of the transmission destination,
Storing the intermediate data in the transmission buffer;
Sorting the array of the intermediate data in the accumulated group of intermediate data based on information that increases the compression rate after sorting either the information of the graph vertex of the transmission source or the information of the graph vertex of the transmission destination And
After the sorting, compress the accumulated group of intermediate data,
A method of transferring intermediate data for graph processing, wherein the group of intermediate data is transferred. The method for transferring intermediate data for graph processing according to claim 1,
Each compute node has a receive buffer,
Each compute node
When the group of transferred intermediate data is sorted based on the information of the source graph vertex,
A method of transferring intermediate data in graph processing, wherein the arrangement of the intermediate data in the group of intermediate data that has arrived at the reception buffer is sorted based on information on the graph vertex of the transmission destination. The method for transferring intermediate data for graph processing according to claim 1,
The method for transferring intermediate data of graph processing, wherein the computing node is a server device. A method for transferring intermediate data in graph processing by a plurality of computation nodes,
Each compute node has a send buffer,
Each compute node is assigned a graph vertex to be processed,
The intermediate data includes a set of information on the graph vertex of the transmission source, information on the graph vertex of the transmission destination, and information on dependency between the graph vertex of the transmission source and the graph vertex of the transmission destination,
Storing the intermediate data in the transmission buffer;
Among the group of accumulated intermediate data based on the information that the compression rate becomes the highest after sorting, either the information of the graph vertex of the transmission source, the information of the graph vertex of the transmission destination, or the information of the dependency relationship Sort the sequence of the intermediate data of
After the sorting, compress the accumulated group of intermediate data,
A method of transferring intermediate data for graph processing, wherein the group of intermediate data is transferred. The method for transferring intermediate data for graph processing according to claim 4 ,
Each compute node has a receive buffer,
Each compute node
When the group of transferred intermediate data is sorted based on the information of the source graph vertex or the dependency information,
A method of transferring intermediate data in graph processing, wherein the arrangement of the intermediate data in the group of intermediate data that has arrived at the reception buffer is sorted based on information on the graph vertex of the transmission destination. The method for transferring intermediate data for graph processing according to claim 4 ,
The method for transferring intermediate data of graph processing, wherein the computing node is a server device. A graph processing system having a plurality of computation nodes,
Each compute node
A transmission buffer for storing intermediate data of graph processing having a pair of information on the graph vertex of the transmission source and information on the graph vertex of the transmission destination;
On the basis of the compression ratio becomes higher information after any sort of information the information or graph vertices of the destination of the transmission source graph vertices, the intermediate of the group of the intermediate data stored in the transmission buffer A module to sort the data sequence;
A graph processing system comprising: a module for compressing the group of intermediate data stored after the sorting; and a module for transferring the group of intermediate data. The graph processing system according to claim 7 ,
Each compute node
A receive buffer;
A graph processing system comprising: a module that sorts the arrangement of the intermediate data in the group of intermediate data that has arrived at the reception buffer based on the information on the graph vertex of the transmission destination. The graph processing system according to claim 7 ,
The graph processing system, wherein the computation node is a server device.

Images