CN115146112A

CN115146112A - Graph calculation method, device, calculation node and storage medium

Info

Publication number: CN115146112A
Application number: CN202110349251.0A
Authority: CN
Inventors: 田腾; 金西; 张学仓
Original assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Current assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2022-10-04

Abstract

The application provides a graph calculation method, a graph calculation device, a calculation node and a storage medium, and belongs to the technical field of graph calculation. The method is applied to a computing node, and comprises the following steps: loading a first set of data from a first storage area of a compute node, wherein the first set of data includes data of a target object indicated by a first subgraph, the first subgraph is a portion of graph data to be computed, and the graph data includes at least one subgraph; loading a second set of data from a second storage area of the compute node, wherein the second set of data includes data of neighbor objects required to update the target object indicated by the first subgraph, the neighbor objects having an associative relationship with the target object, the second set of data being a portion of the graph data; the first set of data is computationally updated based on the second set of data to obtain a first computational result. By the aid of the method and the device, time for loading data can be shortened, and graph calculation efficiency is improved.

Description

Graph calculation method, device, calculation node and storage medium

Technical Field

The present application relates to the field of graph computation technologies, and in particular, to a graph computation method and apparatus, a compute node, and a storage medium.

Background

A graph is a data structure made up of objects, which may also be referred to as vertices, and edges by which the objects are associated. In the fields of social networks, knowledge graphs, recommendation systems and the like, the importance of the objects in the graphs is calculated through the graphs, the similarity between the objects in the graphs is calculated, and the like, so that intelligent recommendation and the like can be achieved. Specifically, in the graph calculation process, the data of the target object is updated by using the neighbor object data of the target object and the like, and the update data of the target object is obtained.

In the related art, graph computation is performed using a computation node, and the computation node loads graph data of a graph into a cache area and then performs computation on the graph data. However, as the attribute dimension of the objects in the graph increases, more storage resources are required to store the data of a single object, and therefore, only a small amount of data of the target object and data of the neighbor objects of the target object can be stored in the cache region of the computing node.

In this way, when graph calculation is performed, the number of times of loading data is large, so that the time taken to load data is large, and graph calculation efficiency is low.

Disclosure of Invention

The application provides a graph calculation method, a graph calculation device, a calculation node and a storage medium, which are used for reducing the time of data loading and improving the efficiency of graph calculation.

In a first aspect, the present application provides a graph computation method, which is performed by a compute node, and includes: loading a first set of data from a first storage area of a compute node, wherein the first set of data includes data of a target object indicated by a first subgraph, the first subgraph is a portion of graph data to be computed, and the graph data includes at least one subgraph; loading a second set of data from a second storage area of the compute node, wherein the second set of data includes data of neighbor objects required to update the target object indicated by the first subgraph, the neighbor objects having an associative relationship with the target object, the second set of data being a portion of graph data; the first set of data is computationally updated based on the second set of data to obtain a first computational result.

In the scheme shown in the application, the graph calculation method can be executed by the calculation node, the graph comprises a plurality of objects, and each object is possibly a target object and also possibly a neighbor object of the target object. The data of a plurality of objects are the graph data to be calculated. The graph data is divided into a plurality of subgraphs, each subgraph indicates data of a target object, and the data of the target object indicated by the first subgraph is loaded to the first storage area, namely the first group of data is loaded to the first storage area. Data of the neighbor objects needed for updating the target object indicated by the first subgraph is loaded to the second storage area, i.e. the second set of data is loaded to the second storage area. When graph computation is performed, the computation node loads a first group of data from the first storage area and loads a second group of data from the second storage area, and computation updating is performed on the first group of data by using the second group of data to obtain a first computation result of the first subgraph. According to the embodiment of the invention, the graph data is divided into the corresponding sub-graphs according to the size of the storage area, and the data of the target object indicated by the sub-graphs and the data of the neighbor object of the target object indicated by the sub-graphs are respectively loaded into different storage areas, so that more data of the target object or more data of the neighbor object can be loaded in the first storage area and the second storage area at one time, and when graph calculation is carried out, the times of loading the data are reduced, the data loading time is saved, and the graph calculation efficiency can be improved.

In one possible implementation, the at least one subgraph is obtained by dividing according to the capacity of the first storage area. In this way, the first storage area is made available for storing data of the target object indicated by the at least one sub-graph.

In one possible implementation, the first storage area is a cache of the second storage area. The first storage area is a cache area and the second storage area is a random access memory, which may also be referred to as a memory.

In one possible implementation, the size of the first sub-graph is equal to the size of the first storage area, and the method further includes: and writing the first calculation result into the second storage area.

According to the scheme, the size of the first sub-graph is equal to the data size of the target object indicated by the first sub-graph, and the size of the first storage area indicates the capacity of the first storage area. When the capacity of the first storage area can just store the data of the target object indicated by the first sub-graph, the computing node can write the first computation result into the second storage area, so that the first storage area does not store the updated data of the target object, and therefore, more objects can be indicated by each sub-graph.

In one possible implementation, the size of the first sub-graph is smaller than the size of the first storage area, and the method further includes: and caching the first calculation result in a first storage area.

According to the scheme, the size of the first sub-graph is smaller than the data size of the target object indicated by the first sub-graph, and the size of the first storage area indicates the capacity of the first storage area. When the capacity of the first storage area can store the data of the target object indicated by the first sub-graph and the update data of the target object indicated by the first sub-graph, the computing node may write the first computation result into the first storage area, where the update data refers to data obtained by performing update computation on the data of the target object. In this way, the update data of the target object can be quickly read and written.

In one possible implementation, the second storage area is a cache of the first storage area. The second storage area is a cache area, and the first storage area is a random access memory, which may also be referred to as a memory.

In one possible implementation manner, the graph calculation includes a plurality of iterative calculations, and the first calculation result is a calculation result obtained by the ith calculation; the first group of data in the (i + 1) th calculation is composed of data of target objects indicated by a plurality of sub-graphs, the number of the plurality of sub-graphs included in the first group of data in the (i + 1) th calculation is determined according to the (i + 1) th calculation and the change of the data amount of each object in the (i) th calculation, or the number of the plurality of sub-graphs included in the first group of data in the (i + 1) th calculation is determined according to the capacity of the first storage area and the data amount of each object in the (i + 1) th calculation.

According to the scheme, graph calculation comprises multiple iterative calculations, the first calculation result is assumed to be the calculation result obtained by the ith calculation, the attribute dimension of an object in the ith calculation is divided by the attribute dimension of the object in the ith +1 calculation, the variation of the data quantity of each object in the ith calculation and the ith calculation is determined, the variation is rounded downwards to obtain a value L, and the value L is multiplied by the number of subgraphs included in the first group of data in the ith calculation, namely the number of subgraphs included in the first group of data in the ith +1 calculation. Or, the data volume of each object in the (i + 1) th calculation is determined by using the attribute dimension of each object in the (i + 1) th calculation. Then, the capacity of the first storage area is divided by the data amount to obtain a value, and the value is rounded down to obtain the number of objects indicated by the i +1 th computation time graph. And then dividing the number by the number of the objects indicated by the subgraphs in the first group of data in the ith calculation, and rounding down to obtain a numerical value, wherein the numerical value is the numerical value L, and the numerical value L is multiplied by the number of the subgraphs included in the first group of data in the ith calculation, namely the number of the subgraphs included in the first group of data in the ith +1 calculation.

Therefore, when the graph calculation has multiple iterative calculations, if the graph calculation is finished each time, the graph data is divided again, extra overhead of the graph data division is undoubtedly brought, however, the graph calculation can be rapidly performed by adaptively adjusting the number of sub-graphs loaded each time without dividing the graph data again, and the graph calculation efficiency can be improved.

In a possible implementation manner, the second set of data includes data of partial neighbor objects required for updating the target object indicated by the first subgraph, the computing node further includes a third storage area, the graph data is stored in the third storage area, and after the first set of data is computationally updated based on the second set of data to obtain the first computation result, the method further includes: loading a third set of data from the third storage area to the second storage area, wherein the third set of data includes data of neighbor objects required for updating the target object indicated by the first sub-graph, and data of neighbor objects in the third set of data does not participate in updating the first set of data; loading a third set of data from the second memory region; and updating the data of the target object indicated by the first subgraph based on the third group of data to obtain a second calculation result.

In the solution presented in the present application, the second set of data comprises data of part of the neighbor objects needed to update the target object indicated by the first sub-graph, so that there may be also neighbor objects that do not participate in updating the first set of data. The computing node further comprises a third storage area, the graph data are stored in the third storage area, and the third storage area is a nonvolatile memory and is large in storage capacity. The computing node may determine whether a third set of data further exists in the third storage area, where the third set of data includes data of neighbor objects required to update the target object indicated by the first sub-graph, and data of the neighbor objects in the third set of data does not participate in updating the first set of data. And if the third group of data exists in the third storage area, loading the third group of data into the second storage area and loading the third group of data from the second storage area. The computing node updates data of the target object indicated by the first subgraph using the third set of data to obtain a second computation result. In this way, in the process of one computation, for each subgraph, only the data of the object indicated by the subgraph is read from the nonvolatile memory once without repeated reading, and then the updated data is not required to be repeatedly written into the nonvolatile memory, so that the read-write amplification of the nonvolatile memory can be reduced.

In a second aspect, the present application provides a graph computation apparatus, which is applied to a computation node, and includes one or more modules, configured to implement the graph computation method described in the first aspect.

In a third aspect, the present application provides a compute node for graph computation, the compute node comprising a processor and a memory, wherein: the memory has stored therein computer instructions that are executed by the processor to cause the computing node to perform the method of the first aspect and possible implementations thereof.

In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions that, when executed by a computing node, cause the computing node to perform the method of the first aspect and possible implementations thereof, or cause the computing node to implement the functions of the apparatus of the second aspect and possible implementations thereof.

In a fifth aspect, the present application provides a computer program product comprising instructions which, when run on a computing node, cause the computing node to perform the method of the first aspect and possible implementations thereof or cause the computing node to implement the functionality of the apparatus of the second aspect and possible implementations thereof.

Drawings

FIG. 1 is a schematic illustration of a graph provided by an exemplary embodiment of the present application;

FIG. 2 is a schematic illustration of graph partitioning provided by an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of a graph computing system provided by an exemplary embodiment of the present application;

FIG. 4 is a block diagram of a compute node provided by an exemplary embodiment of the present application;

FIG. 5 is a diagram of graph data partitioning provided by an exemplary embodiment of the present application;

FIG. 6 is a diagram of graph data partitioning provided by an exemplary embodiment of the present application;

FIG. 7 is a flow chart diagram of a graph computation method provided in an exemplary embodiment of the present application;

FIG. 8 is a block diagram of graph computation provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of a computational update of data of a target object provided by an exemplary embodiment of the present application;

FIG. 10 is a block diagram of graph computation provided in an exemplary embodiment of the present application;

FIG. 11 is a block diagram of graph computation provided by an exemplary embodiment of the present application;

FIG. 12 is a diagram illustrating loading subgraphs and sub-blocks as provided by an exemplary embodiment of the present application;

FIG. 13 is a block diagram of graph computation provided in an exemplary embodiment of the present application;

FIG. 14 is a block diagram of a graph computing device provided in an exemplary embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, the following detailed description of the embodiments of the present application will be made with reference to the accompanying drawings.

To facilitate an understanding of the embodiments of the present application, the following first introduces concepts of the terms involved:

a graph is a data structure consisting of objects and edges. As shown in fig. 1, the graph includes 10 objects, 14 edges. An object may also be referred to as a vertex. Objects may be divided into target objects, which may be referred to as target vertices, and neighbor objects, which may also be referred to as associated objects, source vertices, etc. The target object is an object pointed by an edge in the image, the neighbor object does not exist independently, the target object and the neighbor object have an edge, the edge points to the target object, and the edge reflects the association relationship between the target object and the neighbor object. In the graph calculation process, the data of the target object is updated by using the data of the target object and the data of the neighbor objects of the target object. Optionally, when updating the data of the target data, the data of the edges of the neighbor object and the target object may also be used.

A Graph Neural Network (GNN) is a neural network that runs directly on a data structure consisting of objects and edges, called a graph. By applying the deep learning technology to the data structure such as the graph, the GNN has good application effects in the fields of social networks, knowledge graphs, recommendation systems and the like.

And in the graph calculation, the importance of the objects in the graph is calculated in a graph traversal mode, or the similarity between the objects in the graph is calculated, and the like. For example, when applied in an object classification scenario, some objects in the graph are labeled with labels, and each object v in the graph uses its feature X _v Is shown, andwith tag T of marked object _v In association, the goal of graph computation is to predict the labels of unlabeled objects using labeled objects. As another example, for the Cora dataset in GNNs, the Cora dataset includes 2708 sample points, each sample point being a scientific article, all sample points being classified into 7 classes, the 7 classes being 1) case-based; 2) A genetic algorithm; 3) A neural network; 4) A probabilistic method; 5) Reinforcement learning; 6) Learning rules; 7) Theory. Each paper is represented by a vector with 1433 dimensions, each vector represents a word, the value of each vector represents whether the word indicated by the vector appears in the paper, if the word appears in the paper, the value of the vector is 1, otherwise, the value of the vector is 0. Each paper refers to at least one other paper or is referred to by other papers, namely, the sample points have connection, namely, edges, and no sample point is completely disconnected with other sample points. If the sample point is viewed as an object in the graph, then this is a connected graph, and no isolated object exists. A training set in the Cora dataset is then used to train to obtain a model that is used to predict the class to which the paper represented by each subject in the test set belongs.

In the graph calculation, it is possible to obtain a final result by a plurality of iterative calculations, an output of the ith calculation is used as an input of the (i + 1) th calculation, and the attribute dimension of the object at the ith calculation is higher than that at the (i + 1) th calculation. For example, for a GNN of two layers, two iterative calculations are performed when performing graph calculation, and as shown in table one, for the Cora dataset of GNNs, the attribute dimension of an object at the time of the first calculation is 1433 dimensions, the attribute dimension of an object in the output of the first calculation is 16 dimensions, the attribute dimension of an object at the time of the second calculation is 16 dimensions, and the attribute dimension of an object in the output of the second calculation is 7 dimensions. For the Citeser dataset of GNN, the attribute dimension of the object is 3703 dimensions for the first calculation, the attribute dimension of the object in the output of the first calculation is 16 dimensions, the attribute dimension of the object in the second calculation is 16 dimensions, and the attribute dimension of the object in the output of the second calculation is 6 dimensions. For the Pubmed dataset of GNN, the attribute dimension of the object in the first calculation is 500 dimensions, the attribute dimension of the object in the output of the first calculation is 16 dimensions, the attribute dimension of the object in the second calculation is 16 dimensions, and the attribute dimension of the object in the output of the second calculation is 3 dimensions. For the Reddit dataset of GNN, the attribute dimension of the object at the first calculation is 602 dimensions, the attribute dimension of the object at the output of the first calculation is 64 dimensions, the attribute dimension of the object at the second calculation is 64 dimensions, and the attribute dimension of the object at the output of the second calculation is 41 dimensions.

Watch 1

In the graph calculation, the above is described as multiple iterative calculations, which may also be referred to as multi-layer calculations in some cases, and it is equivalent to include n layers of calculations in the graph calculation process, where the output of the i-th layer of calculation is used as the input of the i + 1-th layer of calculation, and the final output result is the output of the n-th layer.

It should be noted here that, in the graph calculation, the number of objects in the graph does not change every time of calculation, and the edges in the graph do not change, and only the data of the target object in the graph is changed.

In the related art, when a compute node performs graph computation, the compute node loads graph data of a graph into a cache (cache), and then performs computation on the graph data. However, as the attribute dimensions of the objects in the graph increase, more space is needed to store data of a single object, so that only part of graph data can be stored in a cache region of a compute node, which results in that the compute node cannot perform graph computation, and therefore, a graph computation method when the attribute dimensions of the objects are relatively large needs to be provided. In addition, in order to support large-scale graph computation in a practical application scenario and limit the capacity of a cache region of a single computing node, the traditional graph computation needs to divide blocks by means of dividing blocks, and the graph computation is performed on a stand-alone system or a distributed graph computing system. As shown in FIG. 2, the graph is divided into three blocks such that the data of the target object indicated by the blocks and the data of the neighbor objects of the target object may both be stored in a cache area of a stand-alone system or on a single compute node of a distributed graph computing system. However, as the attribute dimensions of objects in graph data increase, more space is required to store data for a single object, and therefore, only data for a limited number of objects can be cached in the cache region of a compute node. Thus, during graph computation, the number of objects in the partitioned blocks is small, edge data corresponding to the objects in the blocks are unevenly distributed, and further, the computation amount of the blocks processed by the computation nodes each time is different, or the load of each computation node in the distributed graph computation system is unbalanced. In addition, when the computing node includes a plurality of computing units, the cache area corresponding to each computing unit stores one block, and the load of each computing unit in the computing node is unbalanced due to uneven distribution of edge data corresponding to objects in the block. Here, the computing unit is a minimum unit for performing graph computation on a computing node, such as a core of a Central Processing Unit (CPU). In addition, when the number of objects in a block is small, the number of times of loading the block is large, which results in a large amount of time spent on loading data, and thus the graph calculation efficiency is low.

Based on this, the present application provides a graph computation method. The method may be applied to the graph computing system 100, and the graph computing system 100 may be a general purpose computer system for graph computing, a dedicated accelerator system, and a programmable hardware system, and may be implemented in software or hardware. The graph computing system can be a stand-alone system including one computing node or a distributed system including a plurality of computing nodes.

Optionally, as shown in fig. 3, from the perspective of logical functions, the graph computing system 100 may include a graph dividing module 101, a processing module 102, and a storage module 103, where the graph dividing module 101 is configured to divide and store graph data into the storage module 103, the processing module 102 is configured to load the graph data and perform graph computation based on the graph data, and the storage module 103 is configured to store the graph data.

For a general-purpose computer system, the processing module 102 may be composed of a general-purpose CPU and a Graphics Processing Unit (GPU). For application specific acceleration systems, the processing module 102 includes an Application Specific Integrated Circuit (ASIC). For a programmable hardware system, the process module 102 includes programmable logic devices, such as a field-programmable gate array (FPGA), etc. For the above three systems, the storage module 103 includes cache areas of the CPU and the GPU, a random access memory, a nonvolatile memory, and the like. The cache region may also be referred to as an on-chip Static Random Access Memory (SRAM) cache, an on-chip Memory region, an on-chip Memory unit, or the like, the random access Memory may be referred to as an off-chip Memory, and the random access Memory may be referred to as a Double Data Rate (DDR) or a High Bandwidth Memory (HBM), or the like. The nonvolatile memory may be a Solid State Disk (SSD), a Hard Disk Drive (HDD), a Solid State Drive (SSD), or the like. For the above three systems, the graph partitioning module 101 is controlled by the processing module 102, and may be implemented by hardware or software.

Alternatively, when the graph computing system 100 is a distributed graph computing system, a plurality of computing nodes included in the distributed graph computing system may share the graph partitioning module 101 and the storage module 103. The processing module 102 may be divided into a control portion and a processing portion, and the control portion of the plurality of computing nodes may be shared, and the processing portions of the plurality of computing nodes may be individually owned. Wherein the control section is used for controlling loading of the map data and the like, and the processing section is used for updating the data of the target object.

Optionally, when the graph computing system 100 is a computing node, as shown in fig. 4, the computing node may include a memory 201 and a processor 202. The memory 201 may include a cache area, a random access memory, and a non-volatile memory, and is used for storing the map data. The processor 202 may be a CPU, microprocessor, ASIC, GPU, or one or more integrated circuits. The processor 202 includes a control unit 2021, a calculation unit 2022, and a graph dividing module 2023, and the control unit 2021 is configured to control the graph dividing module 2023 to divide graph data and may control reading and writing of data by the calculation unit 2022. The computing unit 2022 is used for performing graph computation, and is a minimum unit for performing graph computation on a computing node, such as a core of a CPU. Illustratively, when a general-purpose computer system serves as a computing node, the processing module 102 plays roles of the control unit and the computing unit, and the graph dividing module is implemented by software, divides graph data, and stores the divided graph data in the memory. In the case of a dedicated acceleration system or a programmable hardware system as a compute node, the control unit, the compute unit, the cache area, and the graph partitioning module are all implemented in the processor 202, where the graph partitioning module may be implemented by a hardware functional module or by a software refresh mode for the control unit.

Optionally, the computing node may further comprise a communication interface 203 and a bus 204, not shown in fig. 4. The memory 201, the processor 202 and the communication interface 203 are connected to each other through a bus 204. The communication interface 203 enables communication between the computing node and other devices or communication networks using transceiver modules such as, but not limited to, transceivers. For example, the map data may be acquired through the communication interface 203. Bus 204 may include a path that transfers information between the various components of the compute node.

Alternatively, the graph calculation method in the present application can be applied to a computer system which needs to process graph data. For example, a GNN-based recommendation system, social network analysis, financial wind control, traffic prediction and the like are accelerated in a cloud computing platform, GNN-based protein simulation, molecular chemistry analysis, life science research and the like are accelerated in a scientific research super-computing platform, and GNN-based natural language processing and computer vision are accelerated in a personal terminal platform.

Before describing the graph calculation method, the process of dividing graph data is described first:

the graph division mode is divided into two cases, one case is as follows: the first storage area is a cache of the second storage area, the first storage area is a cache area, and the second storage area is a random access memory (ram). The other condition is that: the second storage area is a cache of the first storage area, the second storage area is a cache area, and the first storage area is a random access memory, which can also be called a memory.

In the embodiment of the present application, the graph computing system is taken as a stand-alone system for explanation, that is, the graph computing system is a single computing node.

The graph partitioning method is described as follows when the first storage area is a cache of the second storage area:

first, the computing node divides the graph data into a plurality of subgraphs using the capacity of the first storage area, so that the first storage area can store data of the target object indicated by at least one subgraph. Suppose that the computing node includes K computing units, K is greater than or equal to 1, each computing unit corresponds to a cache region, and the cache regions corresponding to the K computing units constitute a first storage area.

In a possible implementation manner, in the first calculation of the graph, the calculation node determines the attribute dimension of an object in graph data to be calculated, and the data volume of each object is obtained by multiplying the attribute dimension of the object by the data volume of each attribute dimension. Then, the capacity of the cache region corresponding to each computing unit is divided by the data amount of each object to obtain a numerical value, and the numerical value is rounded down to determine the number of the objects indicated by the divided sub-blocks. The number is then used to divide the graph data into N sub-partitions, N being equal to the rounded-up result for a first value equal to the number of vertices in the graph data divided by the number of objects indicated by each sub-partition, each sub-partition indicating data of a target object. In this way, the cache region of each computing unit stores the data of the object indicated by one sub-partition, K computing units correspond to K cache regions, K sub-partitions can be stored in total, and the K sub-partitions are regarded as one sub-graph, so that the graph data can be divided into a plurality of sub-graphs, and each sub-graph comprises K sub-partitions. The division is equivalent to that the first storage area only stores the data of the target object indicated by the sub-graph, but not stores the update data of the target object indicated by the sub-graph, wherein the update data is obtained after the target data is subjected to calculation and update. In this case, since the second storage area is used as a memory and has a larger storage capacity than the first storage area, the update data of the target object can be stored in the second storage area.

In another possible implementation manner, in the first calculation of the graph, the calculation node determines the attribute dimension of an object in the graph data to be calculated, and the data volume of each object is obtained by multiplying the attribute dimension of the object by the data volume of each attribute dimension. The compute node then multiplies the data volume of the object by 2 to obtain a product. Dividing the capacity of the cache region corresponding to each computing unit by the product to obtain a value, rounding the value down, and determining the number of the objects indicated by the sub-blocks obtained by dividing. The number is then used to divide the graph data into M sub-tiles, M being equal to the rounding up result for a second value, the second value being equal to the number of vertices in the graph data divided by the number of objects indicated by each sub-tile, each sub-tile indicating data of a target object. In this way, the cache region of each computing unit stores one sub-partition, K computing units correspond to K cache regions, K sub-partitions can be stored in total, and K sub-partitions are regarded as one subgraph, so that the graph data can be divided into a plurality of subgraphs, each subgraph including K sub-partitions. This division corresponds to the first storage area storing the data not updated by the target object and the update data. Here, since the attribute dimension of the target object is not changed in the calculation process, the dimension of the updated data of the target object is the same as that of the non-updated data. In the embodiment of the present application, in a calculation update, the data of the target object that is not updated refers to the initial data of the target object at the time of the calculation update.

The capacity of the first storage area is a capacity that can actually store map data.

Secondly, the computing node divides the neighbor objects of the target object in the graph data, namely the computing node divides the graph data to obtain sub-blocks:

a portion of the capacity in the second storage area is used for storing non-updated data and updated data for the target object indicated by the sub-graph. The computing node determines a remaining capacity of the second storage area other than the portion of the capacity. And then the calculation node determines the attribute dimension of each object in the graph data to be calculated in the first calculation process, and the data volume of each object is obtained by multiplying the attribute dimension of each object by the data volume of each attribute dimension. Dividing the remaining capacity by the data volume of each object to obtain a value, rounding the value downwards to determine the number of the objects indicated by the sub-blocks obtained by dividing, dividing the graph data into H sub-blocks according to the number, wherein H is equal to an upward value result of a third value, the third value is equal to the number of top points in the graph data divided by the number of the objects indicated by each sub-block, and the H sub-blocks are used for calculating and updating the data of the target object indicated by the sub-graph. The subsequent second set of data may or may not be equal to the data of the object indicated by one sub-block, which will be described separately later.

Thus, after the graph data is divided into the subgraph and the sub-blocks according to the dividing method, the subgraph indicates the target object, and the sub-blocks indicate the neighbor objects of the target object, then the data of the edges in the graph data is divided according to the objects to which the data belongs, namely, each subgraph and each sub-block correspond to an edge block. For example, for sub-graph I _i And sub-block T _j (iii) sub-diagram I _i And sub-block T _j The corresponding edge block includes sub-graph I _i Target object and sub-block T in (1) _j Data of edges of neighbor objects of the target object. FIG. 5 shows a diagram of a partitioned graph data, wherein I ₁ To I _Q A representation subgraph, representing the existence of Q subgraphs, T ₁ To T _O Indicating a sub-block, indicating that there are O sub-blocks. Subfigure I _i And sub-block T _j Corresponding side block IT _ij E.g. IT ₁₁ Represents sub-graph I ₁ And sub-block T ₁ Corresponding edge block, let K be greater than 1, edge block IT _ij Comprises K sub-edge blocks, each sub-edge block is a sub-image I _i Neutron blocking and sub-blocks T _j Corresponding sub-edge partition, denoted IT _ij-k And K is 1 to K.

The capacity of the second storage area is a capacity capable of storing map data. When the capacity of the second storage area is enough to store the graph data to be calculated, the graph data does not need to be partitioned; when the capacity of the second storage area is not enough to store the graph data to be calculated, the graph data needs to be partitioned.

It should be noted that, whether a subgraph or a sub-block is divided, the graph data is divided, because each object in the graph data may be a target object or a neighbor object of the target object, but the graph data is divided according to different rules, the subgraph is divided according to the capacity of the first storage area, and the sub-block is divided according to the capacity of the second storage area.

The graph partitioning method is described as follows when the second storage area is a cache of the first storage area:

firstly, a neighbor object of a target object in graph data is divided by a computing node, namely, the computing node divides the graph data to obtain sub-blocks:

suppose that the computing node includes K computing units, K is greater than or equal to 1, each computing unit corresponds to a cache region, and the cache regions corresponding to the K computing units constitute a second storage area. Before the first calculation of the graph, the calculation node determines the capacity of a cache region of each calculation unit, determines the attribute dimension of an object in graph data to be calculated, and multiplies the attribute dimension of the object by the data volume of each attribute dimension to obtain the data volume of each object. Then dividing the capacity of the cache area of each computing unit by the data amount of each object to obtain a value, rounding down the value to obtain the number of objects indicated by the divided sub-blocks, dividing the graph data into F sub-blocks using the number, F being equal to the result of rounding up of a fourth value, wherein the fourth value is equal to the number of vertices in the graph data divided by the number of objects indicated by each sub-block, each sub-block indicating one or more objects. Thus, K computing units have K cache banks corresponding thereto, and may store data of an object indicated by K sub-blocks in total, where the K sub-blocks are regarded as one sub-block. Thus, the graph data may comprise at least one sub-block, the data of the object indicated by the at least one sub-block being used to compute data for updating the target object indicated by the sub-graph.

Secondly, the computing node divides the target object in the graph data, namely the computing node divides the graph data to obtain a subgraph:

the capacity of the first storage area is used for storing data of an object indicated by one sub-block. The computing node determines a remaining capacity of the first storage area, the remaining capacity being a capacity of the available capacity in the first storage area other than a capacity occupied by the stored one of the sub-blocks. The calculation node determines the attribute dimension of the object in the graph data to be calculated, determines the data volume of each object by using the attribute dimension of the target object, and then multiplies the data volume of each object by 2 to obtain a product. And dividing the remaining capacity by the product to obtain a value, and rounding the value downwards to determine the number of the objects indicated by the sub-graph obtained by dividing. The number is then used to divide the graph data into S subgraphs, S being equal to the rounded up result for a fifth value, where the fifth value is equal to the number of vertices in the graph data divided by the number of objects indicated by each subgraph, each subgraph indicating data of the target object. Thus, the first storage area stores data corresponding to the object indicated by one sub-block, and data indicating that the target object indicated by one sub-picture is not updated and updated data.

Thus, after the subgraph and the sub-blocks are divided according to the dividing method, the subgraph indicates the target object, the sub-blocks indicate the neighbor objects of the target object, and then the data of the edges in the graph data are divided according to the objects to which the data belong, namely, each subgraph and each sub-block correspond to an edge block. FIG. 6 shows a diagram of a partitioned graph data, wherein I ₁ To I _m A representation subgraph, representing the existence of m sub-graphs, T ₁ To T _p Indicating a sub-block, indicating that there are p sub-blocks. Subfigure I _i And sub-block T _j Corresponding side block TI _ij Such as TI ₁₁ Represents sub-graph I ₁ And sub-block T ₁ Corresponding edge block, let K be greater than 1, edge block TI _ji Including K sub-edge partitions, each sub-edge partition being a sub-image I _i And sub-block T _j The sub-edge sub-block corresponding to the sub-block is denoted as TI _ji-k And K is 1 to K.

The capacity of the first storage area is a capacity that can actually store map data. When the capacity of the first storage area is enough to store the graph data to be calculated, subgraphs do not need to be divided into the graph data, and when the capacity of the first storage area is not enough to store the graph data to be calculated, the subgraphs need to be divided into the graph data.

It should be noted that, no matter the sub-graph or the sub-block is divided, all objects in the graph data to be calculated are divided, because each object in the graph data may be a target object and may also be a neighbor object of the target object, but the division is performed according to different rules, the sub-graph is divided according to the capacity of the first storage area, and the sub-block is divided according to the capacity of the second storage area.

Before the graph data is divided, the graph data is stored in the nonvolatile memory. If the address space of the data of each object in the graph data in the nonvolatile memory is a continuous address space, after the subgraph and the sub-blocks are divided, the computing node records the addresses of the data of the objects indicated by each subgraph in the nonvolatile memory and records the addresses of the data of the objects indicated by each sub-block in the nonvolatile memory. If the address space of the data of each object in the graph data in the nonvolatile memory is not a continuous address space, after the subgraph and the sub-blocks are divided, the computing node loads the data of the object indicated by each subgraph from the nonvolatile memory to the memory, and writes the data into the nonvolatile memory in a re-sequence mode, so that the data of the object indicated by each subgraph are continuously stored in the nonvolatile memory, and loads the data of the object indicated by each sub-block from the nonvolatile memory to the memory, and writes the data into the nonvolatile memory in a re-sequence mode, so that the data of the object indicated by each sub-block are continuously stored in the nonvolatile memory. The computing node records the addresses of the data of the objects indicated by the sub-graphs in the nonvolatile memory, and records the addresses of the data of the objects indicated by the sub-blocks in the nonvolatile memory. In this way, data of an object indicated by one sub-graph can be sequentially read in the continuous address space, and data of an object indicated by one sub-graph can be sequentially read in the continuous address space.

In addition, for any sub-graph and sub-block combination, the computing node reads the data of the edge sub-block corresponding to the combination from the nonvolatile memory to the random access memory, and then writes the data of the edge sub-block corresponding to the combination into the nonvolatile memory in sequence, so that the data of the edge sub-block corresponding to the combination is continuously stored in the nonvolatile memory. And because the edge data has no sequentiality, for one edge block, the edge data in the edge block does not need to be sequenced, and the graph dividing efficiency can be improved. The computing node records the addresses of the edge blocks in the nonvolatile memory, so that the edge data in the edge blocks can be sequentially transmitted instead of random access when being read from the nonvolatile memory.

After the graph data division is performed, the flow of the graph calculation method is described next, and as shown in fig. 7, the processing flow of the graph calculation method is as follows:

step 701, a first set of data is loaded from a first storage area of a compute node, where the first set of data includes data of a target object indicated by a first subgraph, the first subgraph is a part of graph data to be computed, and the graph data includes at least one subgraph.

Step 702, loading a second set of data from a second storage area of the computing node, wherein the second set of data includes data of a neighbor object required for updating the target object indicated by the first subgraph, the neighbor object has an association relationship with the target object, and the second set of data is a part of graph data.

And 703, performing calculation updating on the first group of data based on the second group of data to obtain a first calculation result.

As follows, when describing the graph calculation flow of fig. 7, it is explained by taking an example that a calculation node includes one calculation unit:

in a possible implementation manner, the first storage area is a cache of the second storage area, and the description of the calculation flow for the graph shown in fig. 7 is as follows:

as shown in fig. 8, before starting graph computation, graph data is stored in a non-volatile memory, and a computing node reads a first set of data from the non-volatile memory using an address of a first subgraph, the first set of data including data of a target object indicated by the first subgraph, and writes to a second storage area. Then, the first group of data is read from the second storage area and written into the first storage area. The computing node reads the data of the object indicated by one sub-block, which may be referred to as a first sub-block, from the non-volatile memory using the address of the sub-block, and writes the data of the object indicated by the first sub-block to the second storage area. The data of the object indicated by the first sub-block comprises a second set of data comprising data of neighbor objects needed to update the target object indicated by the first sub-graph. If all objects indicated by the first sub-block are neighbor objects of the target object indicated by the first sub-block, then the objects in the second set of data are the same as the objects indicated by the first sub-block, and if part of the objects indicated by the first sub-block are not neighbor objects of the target object indicated by the first sub-block, then the objects in the second set of data belong to the first sub-block and are less than the objects indicated by the first sub-block.

And then the computing node controls the edge data in the first edge block to be cached from the nonvolatile memory to the second storage area in a streaming manner, wherein the first edge block is the edge block corresponding to the first subgraph and the first sub-block. The computing node controls streaming of the edge data in the first edge chunk from the second storage area to the computing unit of the computing node. It should be noted here that streaming loading refers to: after the edge data of one edge is written into the second storage area, the edge data is transmitted to the computing unit of the computing node for computing, and the computing node deletes the edge data after computing, that is, the edge data in the edge block does not need to be cached completely. Only one computing unit is shown in fig. 8.

And the computing node determines a target object of an edge to which the edge data belongs every time when acquiring the edge data, and randomly reads the data of the target object in the first storage area. The computing node determines another object of the edge to which the edge data belongs, i.e. a neighbor object of the target object. And the computing node sends a reading request of the neighbor object of the target object to the second storage area, and reads the data of the neighbor object of the target object from the data of the object indicated by the first sub-block of the second storage area. It should be noted here that the computing node is a target object and a neighbor object of the target object, which determine an edge to which the edge data belongs, through the control unit.

And the computing unit of the computing node updates the data of the target object by using the data of the neighbor object of the target object, the data of the target object and the edge data to obtain the updated data of the target object. For example, as shown in fig. 9, the target object is B, the neighbor object of the target object B is A, C, and when the data of the target object B is updated by calculation, one way is as follows: if the side data of the AB side is loaded first and the data of the update B is added to the data of B using the data of a, the update data B '= a + B of B, and then the side data of the BD side is loaded again, and the data of the update B is added to the data of D, the update data of B is B "= B' + D based on the DB side. The other mode is as follows: if the edge data of the AB edge is loaded first and the data for updating B is multiplied by the data of B by using the data of A, the updated data of B 'is not equal to A x B, and then the edge data of the BD edge is loaded again, and the data for updating B is multiplied by the data of B and the data of D, the updated data of B is DB based on the DB edge, and then B' obtained based on the AB edge is added with D x B obtained based on the DB edge to obtain the updated data of B. If the subsequent target object B is a neighbor object of another object F, then when the data of another object F is computationally updated, the data of the target object B that is not updated is used. In the above, when the data of the target object is updated, the edge information is considered, where the edge information is used to reflect the weight of the target object by the neighbor object of the target object of the edge, for example, there is an edge between paper 1 and paper 2, where paper 1 is the target object, paper 2 is the neighbor object of the target object, and the edge information is the weight indicating that paper 1 refers to paper 2. In fig. 9, assuming that the weight between a and B is a and the weight between D and B is B, when updating the data of the target object B using a, the update data B '= a' + B of B, and when updating the data of the target object B using D, the update data B '= B' + D of B.

According to the method, after all the edge data in the first edge partition are processed, the first calculation result corresponding to the first subgraph is obtained.

Optionally, after the first calculation result is obtained, the first calculation result may be further stored, and the processing is as follows:

as shown in fig. 10, if the size of the first sub-graph is equal to the size of the first storage area, the first calculation result cannot be written in the first storage area, and the calculation node writes the first calculation result in the second storage area. It should be noted here that, in an actual operation, each time the computing node obtains the update data of the target object indicated by the first sub-graph, the update data may be written into the second storage area, instead of waiting for obtaining the first computation results corresponding to all the target objects indicated by the first sub-graph, and writing the first computation results into the second storage area.

As shown in fig. 11, if the size of the first subgraph is smaller than the size of the first storage area, the first computation result may be written to the first storage area, e.g., if the size of the first subgraph is equal to half the capacity of the first storage area, the computation node may write the first computation result to the first storage area. It should be noted here that, in actual operation, each time the computing node obtains the update data of the target object indicated by the first sub-graph, the update data may be written into the first storage area, instead of waiting for obtaining the first computation results of all the target objects indicated by the first sub-graph, and writing the first computation result into the first storage area.

Optionally, after the first calculation result is obtained, in order to release the memory usage of the second group of data, the second group of data may be deleted.

It should be noted that, in the description of the flow in fig. 7, if the capacity of the second storage area is sufficient to store the graph data to be calculated, the second set of data includes data of all neighbor objects required for updating the target object indicated by the first sub-graph, and if the capacity of the second storage area is insufficient to store the graph data to be calculated, the second set of data includes data of part of neighbor objects required for updating the target object indicated by the first sub-graph, so that the first sub-graph is not repeatedly loaded, and after the first set of data is updated by calculation using the second set of data, the following processing may be performed:

loading a third set of data from the third storage area to the second storage area, wherein the third set of data includes data of neighbor objects required for updating the target object indicated by the first sub-graph, and data of neighbor objects in the third set of data does not participate in updating the first set of data; loading a third set of data from the second memory region; and updating the data of the target object indicated by the first subgraph based on the third group of data to obtain a second calculation result.

The third storage area is the nonvolatile memory described above, has a relatively large capacity, and can store the graph data to be calculated.

In this embodiment, the computing node may determine whether a third set of data still exists in the non-volatile memory, where the third set of data includes data of neighbor objects required to update the target object indicated in the first sub-graph, that is, determine whether a sub-block in the non-volatile memory does not participate in updating the target object indicated in the first sub-graph. If the sub-block exists, the data of the object indicated by the sub-block is loaded from the nonvolatile memory and written into the second storage area, the sub-block is subsequently called as a second sub-block, and the third group of data is the data of the neighbor object required for updating the target object indicated by the first sub-block in the object indicated by the second sub-block. And the computing node controls the streaming of the edge data in the second edge partition block from the nonvolatile memory to a second storage area, wherein the second edge partition block is an edge partition block corresponding to the first subgraph and the second sub-block.

The computing node then controls streaming of the edge data in the second edge partition block from the second storage area to the computing unit of the computing node. And each time the computing node acquires one piece of edge data, determining a target object of the edge to which the edge data belongs, and randomly reading the data of the target object in the first storage area. The computing node determines another object of the edge to which the edge data belongs, namely a neighbor object of the target object, sends a reading request of the neighbor object of the target object to the second storage area, and reads data of the neighbor object of the target object from the data of the objects indicated by the second sub-block of the second storage area. And the computing unit of the computing node updates the data of the target object by using the data of the neighbor object of the target object, the data of the target object and the edge data to obtain the updated data of the target object. Since the first set of data has already been computationally updated using the second set of data, the data of the target object here may include updated data in addition to the non-updated data, so that the non-updated data and the updated data are considered when updating the data of the target object. See the example of fig. 9 for a specific calculation update process.

According to the mode, after all the edge data in the second edge partition block are finished, the second calculation result corresponding to the first subgraph is obtained.

The computing node may store the second computation result, and replace the first computation result with the second computation result when storing the second computation result.

And then, the computing node continuously judges whether a neighbor object which does not participate in updating the first group of data exists in the nonvolatile memory, if so, the process of loading the third group of data is continuously executed, and if not, the second computing result is subjected to linear change, so that the dimensionality of the second computing result is reduced, namely, the attribute dimensionality of the data of the target object indicated by the first sub-graph is reduced. And then the computing node writes the second computing result after the linear transformation into the nonvolatile memory. In this way, after one calculation is performed on the graph data, the attribute dimensions of each object obtained are the same.

In this way, in one computation process, for each subgraph, the data of the object indicated by the subgraph is read only once from the nonvolatile storage area without repeated reading, and the read-write amplification of the nonvolatile memory can be reduced.

The compute node then determines whether there are other subgraphs in the non-volatile memory that are not updated. If yes, continuing to load the sub-graph, and executing the process of updating the data of the target object indicated by the sub-graph; if not, it is determined whether the graph computation process is to perform a computation update only once. If the calculation updating is executed only once, the graph calculation process is determined to be completed; if the iterative computation update is executed for multiple times, executing the next computation update, wherein the data obtained by the last computation update is input for the next computation update, and since the attribute dimension of the object in the data obtained by the last computation update is reduced, the data volume of the target object indicated by the corresponding first sub-graph is reduced. The following are exemplary:

assuming that the previous calculation is the ith calculation and the current calculation is the (i + 1) th calculation, dividing the attribute dimension of the object in the ith calculation by the attribute dimension of the object in the (i + 1) th calculation to determine the variation of the data volume of each object in the (i + 1) th calculation and the ith calculation. And rounding the variation downwards to obtain a value L, and multiplying the value L by the number of subgraphs included in the first group of data in the ith calculation, namely the number of subgraphs included in the first group of data in the (i + 1) th calculation. In addition, for the subblocks, the same principle applies, and the number L is multiplied by the number of subblocks corresponding to the second group of data in the ith calculation, that is, the number of subblocks loaded from the nonvolatile memory to the second storage area at one time in the (i + 1) th calculation. For example, as shown in FIG. 12, the ith calculation loads one sub-block and one sub-graph each time, and the (i + 1) th calculation loads L sub-blocks and L sub-graphs each time, where L is equal to 2 in FIG. 12.

In another possible implementation manner, the above value L may be calculated in another manner, and the processing is as follows:

and determining the data volume of each object in the (i + 1) th calculation by using the attribute dimension of each object in the (i + 1) th calculation. Then, the capacity of the first storage area is divided by the data amount to obtain a value, and the value is rounded down to obtain the number of objects indicated by the graph at the (i + 1) th computation time. And dividing the number by the number of objects indicated by the sub-map in the first group of data in the ith calculation, and rounding down to obtain a numerical value, wherein the numerical value is the numerical value L.

Therefore, when multiple iterative computations exist in the graph computation process, if the computation is completed each time, the graph data is re-divided, and extra overhead of graph data division is undoubtedly brought.

Thus, the random read-write bandwidth of the nonvolatile memory is reduced by 2 orders of magnitude compared with the sequential read-write bandwidth. When the first storage area is the cache of the second storage area, only sequential reading and sequential writing exist in the read-write of the nonvolatile memory according to the modes of subgraphs and subblocks, so random read-write access to the nonvolatile memory is avoided.

And when the compute node includes a plurality of compute units, the plurality of compute units each compute a sub-block in the first subgraph. The plurality of computing units respectively compute data for updating the target objects indicated by the plurality of sub-blocks in the first sub-graph, the plurality of computing units may access the second storage area in multiple threads, and although accessing the second storage area is random reading and writing, however, the efficiency of random reading and writing of the second storage area is equivalent to that of sequential reading and writing, so that the calculation efficiency of the graph cannot be reduced by the random reading and writing, and the reading and writing bandwidth of the second storage area can be fully utilized.

In addition, in the embodiment of the application, only the data of the object indicated by the sub-graph is loaded to the first storage area, and the data indicated by the sub-block is not also loaded to the first storage area, so that a plurality of objects are indicated by a single sub-graph. In this way, when the computing node includes a plurality of computing units, edge data in the sub-blocks of each sub-graph and the edge sub-blocks corresponding to each sub-block are uniformly distributed, so that load of each computing unit is balanced.

In another possible implementation manner, the second storage area is a cache of the first storage area, and the description of the calculation flow for the graph shown in fig. 7 is as follows:

as shown in fig. 13, before starting graph computation, graph data is stored in a non-volatile memory, and a compute node may read a first set of data from the non-volatile memory using an address of a subgraph, the first set of data including data of a target object indicated by the first subgraph, and write to a first storage area. The compute node may use the address of the sub-block to read from non-volatile memory the data of the object indicated by one sub-block, which may be referred to as the first sub-block. And writing the data of the object indicated by the first sub-block into the first storage area. The data of the object indicated by the first sub-block is then written from the first storage area to the second storage area. The data of the object indicated by the first sub-block comprises a second set of data comprising data of neighbor objects needed to update the target object indicated by the first sub-graph. If all objects indicated by the first sub-block are neighbor objects of the target object indicated by the first sub-block, then the objects in the second set of data are the same as the objects indicated by the first sub-block, and if part of the objects indicated by the first sub-block are not neighbor objects of the target object indicated by the first sub-block, then the objects in the second set of data belong to the first sub-block and are less than the objects indicated by the first sub-block.

And then the computing node controls the edge data in the first edge block to be cached from the nonvolatile memory to a first storage area in a streaming manner, the first edge block is the edge block corresponding to the first subgraph and the first sub-block, and the computing node controls the edge data in the target edge block to be transmitted from the first storage area to the computing unit of the computing node in the streaming manner.

And each time the computing node acquires one piece of edge data, determining the target object of the edge to which the edge data belongs, sending a reading request of the target object to the first storage area, and reading the data of the target object from the first group of data in the first storage area. And the computing unit of the computing node determines another object of the edge to which the edge data belongs, namely a neighbor object of the target object, and randomly accesses the second storage area to obtain the data of the neighbor object of the target object. It should be noted here that the computing node is a target object and a neighbor object of the target object, where the control unit determines an edge to which the edge data belongs, and the control unit provides the computing unit with data to be computed.

And the computing unit of the computing node updates the data of the target object by using the data of the neighbor object of the target object, the data of the target object and the edge data, and obtains the updated data of the target object. The process of updating the data of the target object is shown in the example of fig. 9, and is not described herein again.

According to the above manner, after all the edge data in the target edge partition is completed, the first calculation result corresponding to the first group of data, that is, the first calculation result corresponding to the first sub-graph is obtained.

In one possible implementation, after going through the flow of fig. 7, the compute node may write the first compute result to the first storage area. It should be noted here that, in an actual operation, each time the computing node obtains the update data of the target object indicated by the first sub-graph, the update data may be written into the first storage area, instead of waiting for obtaining the first computation results corresponding to all the target objects indicated by the first sub-graph, the first computation result is written into the first storage area.

It should be noted that, when the flow of fig. 7 is described, if the capacity of the first storage area is sufficient to store the graph data to be calculated and the update data of the graph data, the graph data is not divided, and the graph data includes one sub graph, that is, the first sub graph. And if the capacity of the first storage area is not enough to store the graph data to be calculated, dividing the graph data to obtain a plurality of subgraphs, wherein the first subgraph is one of the subgraphs.

It should be noted that, if the capacity of the second storage area is sufficient to store the graph data to be computed, the second set of data includes data of all neighbor objects required to update the target object indicated by the first sub-graph, which generally does not exist because the second storage area is a cache area. If the capacity of the second storage area is not sufficient to store the graph data to be computed, the second set of data comprises data of part of the neighboring objects required for updating the target object indicated by the first sub-graph, and in order to avoid reloading the first sub-graph, after the first set of data is computationally updated using the second set of data, the following process can be performed:

loading a third set of data from the third storage region to the first storage region, wherein the third set of data includes data of neighbor objects required for updating the target object indicated by the first sub-graph, and the data of the neighbor objects in the third set of data does not participate in updating the first set of data; loading a third set of data from the first memory region; and updating the data of the target object indicated by the first subgraph based on the third group of data to obtain a second calculation result.

In this embodiment, the computing node may determine whether a third set of data still exists in the non-volatile memory, where the third set of data includes data of neighbor objects required to update the target object indicated in the first sub-graph, that is, determine whether a sub-block in the non-volatile memory does not participate in updating the target object indicated in the first sub-graph. If such a sub-block exists, the data of the object indicated by the sub-block is loaded from the non-volatile memory and written into the first storage area, and the sub-block is subsequently referred to as a second sub-block, and then the data of the object indicated by the second sub-block is written into the second storage area. And the computing node controls the streaming of the edge data in the second edge partition block from the nonvolatile memory to the first storage area, wherein the second edge partition block is an edge partition block corresponding to the first subgraph and the second sub-block. And controlling the streaming transmission of the edge data in the second edge partition block from the first storage area to the computing unit of the computing node.

When the computing node acquires one piece of edge data, the target object of the edge to which the edge data belongs is determined. And sending a reading request of the target object to the first storage area, reading data of the target object, and randomly reading data of the neighbor object of the target object from the second sub-block of the second storage area.

And the computing unit of the computing node updates the data of the target object by using the data of the neighbor object of the target object, the data of the target object and the edge data to obtain target object update data. Since the first set of data has already been computationally updated using the second set of data, the data of the target object herein may include updated data in addition to the non-updated data, so that the non-updated data and the updated data are considered when updating the data of the target object. See the example of fig. 9 for a specific calculation update procedure.

The computing node then proceeds to determine whether there are any neighbor objects in the non-volatile memory that are not involved in updating the first set of data. If yes, continuing to execute the process of loading the third group of data; if the first sub-graph does not exist, the second calculation result is linearly changed, and the dimension of the second calculation result is reduced, namely the attribute dimension of the data of the target object indicated by the first sub-graph is reduced. And then the computing node writes the second computing result after the linear transformation into the nonvolatile memory.

And then the computing node judges whether other subgraphs exist in the nonvolatile memory and are not updated, if so, the subgraph is continuously loaded, the flow of updating the data of the target object indicated by the subgraph is executed, and if not, the graph computing process is determined to be that the computation updating is not executed only once. If the calculation updating is executed only once, the graph calculation process is determined to be finished; and if the iterative computation update is executed for multiple times, executing the next computation update, wherein the data obtained by the last computation update is the input of the next computation update. In order to perform graph calculation quickly and not to divide graph data again, the number of sub-graphs loaded into the first storage area at one time at this time can be determined based on the attribute dimension of the object in the last calculation and the attribute dimension of the object in the current calculation, and the number of sub-blocks loaded into the second storage area at one time at this time can be determined. The determination method is described in the foregoing, and is not described in detail here.

By the aid of the method and the device, the subgraph and the sub-blocks are divided independently, and the problem of uneven edge data distribution caused by fewer objects indicated by the subgraph can be solved. After a sub-graph is loaded, the data of the target object indicated by the sub-graph is updated by using all the neighbor objects in the graph data, and then the updated data of the sub-graph is written into the nonvolatile memory, so that the random writing to the nonvolatile memory is avoided.

In the graph calculation process, if multiple iterative calculations exist, the graph calculation can be performed quickly by adaptively adjusting the number of sub graphs and the number of sub blocks loaded each time without repartitioning graph data, and the graph calculation efficiency is improved.

In addition, in the embodiment of the application, only the data of the object indicated by the sub-graph is loaded to the first storage area, and the data indicated by the sub-block is not loaded to the first storage area, so that the number of the objects indicated by a single sub-graph is large. Therefore, more data of the target object and more data of the neighbor object are loaded from the nonvolatile memory at one time, the data loading times are reduced, the data loading time is reduced, and the graph calculation efficiency can be improved. When the computing node comprises a plurality of computing units, the edge data in the edge sub-blocks are uniformly distributed, so that the load of each computing unit can be balanced.

It should be noted that, when the method of graph computation is introduced, the graph computation system is taken as a single computation node for example, and when the graph computation system is a distributed graph computation system, the processing procedure of graph computation is similar to that described in the foregoing, but multiple computation nodes perform graph computation in parallel, and the multiple computation nodes may compute and update data of target objects indicated by multiple subgraphs in parallel. When the graph data is divided, each computing node is equivalent to each computing unit in the foregoing, the sub-blocks in the subgraph corresponding to each computing unit are equivalent to the sub-blocks corresponding to each computing node, the sub-blocks processed by all the computing nodes at one time form one subgraph, and only because the processing capacity of the computing node is higher than that of a single computing unit, the target objects indicated by the sub-blocks in the subgraph corresponding to each computing node are more than the target objects indicated by the sub-blocks in the subgraph corresponding to each computing unit.

Fig. 14 is a block diagram of a device for graph computation provided in an embodiment of the present application. The apparatus may be implemented as part or all of an apparatus, for application to a computing node, by software, hardware, or a combination of both. The apparatus provided in this embodiment of the present application may implement the process described in fig. 7 in this embodiment of the present application, and the apparatus includes: a control module 1410 and a calculation module 1420, wherein:

a control module 1410 to:

loading a first set of data from a first storage area of the compute node, wherein the first set of data includes data of a target object indicated by a first subgraph that is part of graph data to be computed, the graph data including at least one subgraph;

loading a second set of data from a second storage area of the computing node, where the second set of data includes data of a neighbor object required for updating a target object indicated by the first subgraph, the neighbor object having an association relationship with the target object, and the second set of data is a part of the graph data, and may be specifically used to implement the loading function of

steps

701 and 702 and to execute implicit steps included in

steps

701 and 702;

the calculating module 1420 is configured to perform calculation and update on the first group of data based on the second group of data to obtain a first calculation result, and may specifically be configured to implement the updating function of step 703 and execute the implicit step included in step 703.

In a possible implementation manner, the at least one sub-graph is obtained by dividing according to the capacity of the first storage area.

In a possible implementation manner, the first storage area is a cache of the second storage area.

In a possible implementation manner, the size of the first sub-graph is equal to the size of the first storage area, and the calculating module 1420 is further configured to:

and writing the first calculation result into the second storage area.

In a possible implementation manner, the size of the first sub-graph is smaller than the size of the first storage area, and the calculation module 1420 is further configured to cache the first calculation result in the first storage area.

In a possible implementation manner, the second storage area is a cache of the first storage area.

In a possible implementation manner, the graph calculation includes a plurality of iterative calculations, and the first calculation result is a calculation result obtained by the ith calculation; the first set of data in the (i + 1) th computation consists of data of the target object indicated by the multiple subgraphs, where:

the number of a plurality of subgraphs included in the first group of data in the (i + 1) th calculation is determined according to the (i + 1) th calculation and the change of the data quantity of each object in the ith calculation; or,

the number of the plurality of subgraphs included in the first group of data in the (i + 1) th calculation is determined according to the capacity of the first storage area and the data amount of each object in the (i + 1) th calculation.

In one possible implementation, the second set of data includes data of a part of neighbor objects required to update a target object indicated by the first subgraph, the computing node further includes a third storage area, the graph data is stored in the third storage area, and the control module 1410 is further configured to:

loading a third set of data from the third storage region to the second storage region, wherein the third set of data includes data of neighbor objects needed to update a target object indicated by the first subgraph, and data of neighbor objects in the third set of data does not participate in updating the first set of data;

loading the third set of data from the second memory region;

the calculating module 1420 is further configured to update data of the target object indicated by the first sub-graph based on the third set of data to obtain a second calculation result.

The division of the modules in the embodiments of the present application is schematic, and only one logic function division is provided, and in actual implementation, there may be another division manner, and in addition, each functional module in each embodiment of the present application may be integrated in one processor, may also exist alone physically, or may also be integrated in one module by two or more modules. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a terminal device (which may be a personal computer, a mobile phone, or a network device) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The descriptions of the flows corresponding to the above-mentioned figures have respective emphasis, and for parts not described in detail in a certain flow, reference may be made to the related descriptions of other flows.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware or any combination thereof, and when the implementation is realized by software, all or part of the implementation may be realized in the form of a computer program product. The computer program product comprises one or more computer program instructions which, when loaded and executed on a server or terminal, cause the processes or functions described in accordance with embodiments of the application to be performed, in whole or in part. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optics, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium can be any available medium that can be accessed by a server or a terminal or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a Digital Video Disk (DVD), etc.), or a semiconductor medium (such as a solid state disk, etc.).

The above description is only exemplary of the application and should not be taken as limiting the application, and any modifications, equivalents, improvements and the like that are made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A graph computation method, performed by a compute node, the method comprising:

loading a second set of data from a second storage area of the compute node, wherein the second set of data includes data of neighbor objects required to update a target object indicated by the first subgraph, the neighbor objects having an associative relationship with the target object, the second set of data being part of the graph data;

and performing calculation updating on the first set of data based on the second set of data to obtain a first calculation result.

2. The method of claim 1, wherein the first and second light sources are selected from the group consisting of, it is characterized in that the preparation method is characterized in that, the at least one subgraph is obtained by dividing according to the capacity of the first storage area.

3. The method of claim 1 or 2, wherein the first storage area is a cache of the second storage area.

4. The method of any of claims 1 to 3, wherein the size of the first sub-graph is equal to the size of the first storage area, the method further comprising:

and writing the first calculation result into the second storage area.

5. The method of any of claims 1 to 3, wherein a size of the first sub-graph is smaller than a size of the first storage area, the method further comprising:

and caching the first calculation result to the first storage area.

6. The method of claim 1 or 2, wherein the second storage area is a cache of the first storage area.

7. The method according to any one of claims 2 to 6, wherein the graph calculation includes a plurality of iterative calculations, and the first calculation result is a calculation result obtained by the ith calculation; the first set of data in the (i + 1) th computation consists of data of the target object indicated by the multiple subgraphs, where:

8. The method of any of claims 2 to 6, wherein the second set of data includes data of partial neighbor objects required to update the target object indicated by the first sub-graph, wherein the computing node further includes a third storage area, wherein the graph data is stored in the third storage area, and wherein after performing the computational update on the first set of data based on the second set of data to obtain the first computation result, the method further comprises:

loading the third set of data from the second memory region;

updating data of the target object indicated by the first sub-graph based on the third set of data to obtain a second calculation result.

9. A graph computation apparatus, applied to a compute node, the apparatus comprising:

a control module to:

loading a second set of data from a second storage area of the compute node, wherein the second set of data includes data of neighbor objects required to update a target object indicated by the first subgraph, the neighbor objects having an associative relationship with the target object, the second set of data being a portion of the graph data;

and the calculation module is used for performing calculation updating on the first group of data based on the second group of data to obtain a first calculation result.

10. The apparatus of claim 9, wherein the at least one sub-graph is partitioned according to a capacity of the first storage area.

11. The apparatus of claim 9 or 10, wherein the first storage area is a cache of the second storage area.

12. The apparatus of any of claims 9 to 11, wherein a size of the first sub-graph is equal to a size of the first storage area, the computation module further configured to:

and writing the first calculation result into the second storage area.

13. The apparatus of any of claims 9 to 11, wherein the size of the first sub-graph is smaller than the size of the first storage area, and wherein the computation module is further configured to cache the first computation result in the first storage area.

14. The apparatus of claim 9 or 10, wherein the second storage area is a cache of the first storage area.

15. The apparatus according to any one of claims 10 to 14, wherein the graph calculation includes a plurality of iterative calculations, and the first calculation result is a calculation result obtained by the ith calculation; the first set of data in the (i + 1) th computation consists of data of the target object indicated by the multiple subgraphs, where:

16. The apparatus of any of claims 10 to 14, wherein the second set of data includes data for partial neighbor objects needed to update the target object indicated in the first sub-graph, wherein the compute node further includes a third storage area, wherein the graph data is stored in the third storage area, and wherein the control module is further configured to:

loading the third set of data from the second memory region;

the calculation module is further configured to update data of the target object indicated by the first sub-graph based on the third group of data to obtain a second calculation result.

17. A compute node for graph computation, the compute node comprising a processor and a memory, wherein:

the memory having stored therein computer instructions;

the processor executing the computer instructions to cause the compute node to perform the method of any of claims 1 to 8.

18. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a computing node, cause the computing node to perform the method of any of claims 1 to 8.