CN112990332B - Sub-graph scale prediction and distributed training method and device and electronic equipment - Google Patents

Sub-graph scale prediction and distributed training method and device and electronic equipment Download PDF

Info

Publication number
CN112990332B
CN112990332B CN202110328443.3A CN202110328443A CN112990332B CN 112990332 B CN112990332 B CN 112990332B CN 202110328443 A CN202110328443 A CN 202110328443A CN 112990332 B CN112990332 B CN 112990332B
Authority
CN
China
Prior art keywords
nodes
graph
sub
center
group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110328443.3A
Other languages
Chinese (zh)
Other versions
CN112990332A (en
Inventor
方超
罗逸璕
陈国栋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Digital Technology Co Ltd filed Critical Hangzhou Hikvision Digital Technology Co Ltd
Priority to CN202110328443.3A priority Critical patent/CN112990332B/en
Publication of CN112990332A publication Critical patent/CN112990332A/en
Application granted granted Critical
Publication of CN112990332B publication Critical patent/CN112990332B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2323—Non-hierarchical techniques based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the application provides a sub-graph scale prediction and distributed training method, a device and electronic equipment, wherein the number, the average degree and the number of sub-graphs of a graph are obtained, and the average node number of the sub-graphs is obtained by using a prediction model according to the number, the average degree and the number of the sub-graphs; if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is adjusted, and the step of obtaining the average node number of the sub-graph by utilizing a prediction model is performed according to the node number, the average degree and the number of the sub-graph; and if the number of the sub-graph average nodes does not exceed the upper limit of the node number, determining a sub-graph scale prediction result comprising the current number of the sub-graph and the number of the sub-graph average nodes. By the scheme, the possibility that the computing unit can process the sub-graph is increased, so that the computing unit can process the sub-graph normally to a certain extent.

Description

Sub-graph scale prediction and distributed training method and device and electronic equipment
Technical Field
The present disclosure relates to the field of machine learning technologies, and in particular, to a sub-graph scale prediction and distributed training method, apparatus, and electronic device.
Background
In the technical field of machine learning, in order to enable a machine to understand knowledge in certain specialized fields and irregular data structures among entities, related knowledge and/or entities are often described in the form of graphs in graph theory, such as social networks, knowledge graphs, chemical molecular structures, and the like. The graph comprises a plurality of nodes, the nodes represent knowledge and/or entities, the connection lines between the nodes are called edges, and the edges are used for representing the association relationship between the two nodes.
Along with the continuous expansion of knowledge and entity scale and the continuous increase of historical data, the number of nodes and edges in the graph is very huge, and a common computing unit cannot perform normal graph processing due to the limitation of computing resources, so that great challenges are brought to the graph processing.
The best way to solve the above problem is distributed processing, i.e. the graph is cut into a plurality of sub-graphs, and the distributed computing unit processes each sub-graph to achieve the purpose of processing the whole graph. However, in practical applications, the sub-graph size that the computing unit can carry is limited, and if the graph is not reasonably cut, for example, the number of nodes included in each sub-graph is very large, the computing unit cannot process the sub-graph.
Disclosure of Invention
The embodiment of the application aims to provide a sub-graph scale prediction and distributed training method, device and electronic equipment, so that a computing unit can process a sub-graph normally to a certain extent. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a sub-graph scale prediction method, where the method includes:
acquiring the number, average degree and number of sub-graphs of nodes of the graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is the average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of the subgraphs is a set value under the initial condition;
obtaining the average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number; the prediction model is established based on a preset graph cutting algorithm and a graph of the subgraph number to obtain a cutting result of the subgraph;
if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is adjusted, and the step of obtaining the average node number of the sub-graph by utilizing a prediction model is performed according to the node number, the average degree and the number of the sub-graph;
And if the number of the sub-graph average nodes does not exceed the upper limit of the node number, determining a sub-graph scale prediction result, wherein the sub-graph scale prediction result comprises the current number of the sub-graph and the number of the sub-graph average nodes.
Optionally, the step of cutting the graph based on a preset graph cutting algorithm and the number of the subgraph to obtain the subgraph comprises the following steps:
selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs;
and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, for any group of center nodes, based on the neighbor relation of the group of center nodes, expanding and scanning outwards to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges, including:
aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph;
Based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
Optionally, the prediction model is:
Figure BDA0002995460580000021
wherein V is S (A) In order to obtain the average node number of the subgraphs after cutting the graph A, V (A) is the node number of the graph A, C (A) is the subgraph number of the graph A, d (A) is the average degree of the graph A, and gamma 1 For outwards expanding and scanning expansion rate of first-order neighbor nodes based on neighbor relation of center node, gamma 2 And expanding and scanning the expansion rate of the second-order neighbor node for outward expansion based on the neighbor relation of the first-order neighbor node.
Alternatively, the expansion rate gamma 1 And gamma 2 The linear regression model is obtained through fitting, and is established based on the relevance of the expansion rate, the number of nodes and the average degree.
Alternatively, the linear regression model is:
Figure BDA0002995460580000031
γ 2 ＝α*γ 1
wherein omega 1 、ω 2 、ω 3 、ω 4 、ω 5 、ω 6 Is a preset weight; b is a constant; alpha is the relative expansion rate gamma 1 And gamma 2 Performing fine tuning.
In a second aspect, embodiments of the present application provide a distributed training method, including:
Cutting the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain each subgraph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the number of the subgraphs is obtained based on the subgraph scale prediction method provided by the first aspect of the embodiment of the application;
and respectively sending each sub-graph to each computing unit so as to train a preset graph processing algorithm by using each sub-graph in each computing unit.
Optionally, based on a preset graph cutting algorithm and the number of the subgraphs, cutting the graph to obtain each subgraph, including:
selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs;
and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, for any group of center nodes, based on the neighbor relation of the group of center nodes, expanding and scanning outwards to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges, including:
Aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph;
based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
In a third aspect, an embodiment of the present application provides a sub-graph scale prediction apparatus, where the apparatus includes:
the acquisition module is used for acquiring the number, average degree and sub-graph number of the nodes of the graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is the average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of the subgraphs is a set value under the initial condition;
the prediction module is used for obtaining the average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number; the prediction model is established based on a preset graph cutting algorithm and a graph of the subgraph number to obtain a cutting result of the subgraph;
The adjusting module is used for adjusting the number of the subgraphs if the average node number of the subgraphs exceeds the upper limit of the node number which can be borne by the computing unit;
and the determining module is used for determining a sub-graph scale prediction result if the number of the sub-graph average nodes does not exceed the upper limit of the node number, wherein the sub-graph scale prediction result comprises the current number of the sub-graphs and the number of the sub-graph average nodes.
Optionally, the apparatus further comprises:
the prediction model building module is used for selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, the prediction model building module is specifically configured to, for any group of center nodes, based on a neighbor relation of the group of center nodes, perform outward expansion scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determine that the group of center nodes, the first-order neighbor nodes, and edges between the group of center nodes and the first-order neighbor nodes form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
Optionally, the prediction model is:
Figure BDA0002995460580000051
wherein V is S (A) In order to obtain the average node number of the subgraphs after cutting the graph A, V (A) is the node number of the graph A, C (A) is the subgraph number of the graph A, d (A) is the average degree of the graph A, and gamma 1 For outwards expanding and scanning expansion rate of first-order neighbor nodes based on neighbor relation of center node, gamma 2 And expanding and scanning the expansion rate of the second-order neighbor node for outward expansion based on the neighbor relation of the first-order neighbor node.
Alternatively, the expansion rate gamma 1 And gamma 2 The linear regression model is obtained through fitting, and is established based on the relevance of the expansion rate, the number of nodes and the average degree.
Alternatively, the linear regression model is:
Figure BDA0002995460580000052
γ 2 ＝α*γ 1
wherein omega 1 、ω 2 、ω 3 、ω 4 、ω 5 、ω 6 Is a preset weight; b is a constant; alpha is the relative expansion rate gamma 1 And gamma 2 Performing fine tuning.
In a fourth aspect, embodiments of the present application provide a distributed training apparatus, the apparatus comprising:
the cutting module is used for cutting the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain each subgraph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the number of the subgraphs is obtained based on the subgraph scale prediction method provided by the first aspect of the embodiment of the application;
And the sending module is used for respectively sending each sub-graph to each computing unit so as to train the preset graph processing algorithm by using each sub-graph in each computing unit.
Optionally, the cutting module is specifically configured to select multiple groups of center nodes from the nodes according to the number of the subgraphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, the cutting module is specifically configured to, for any group of center nodes, based on a neighbor relation of the group of center nodes, perform outward expansion scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determine that the group of center nodes, the first-order neighbor nodes, and edges between the group of center nodes and the first-order neighbor nodes form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
In a fifth aspect, embodiments of the present application provide an electronic device including a processor and a memory;
a memory for storing a computer program;
and the processor is used for realizing the sub-graph scale prediction method provided by the first aspect of the embodiment of the application or the distributed training method provided by the second aspect of the embodiment of the application when executing the computer program stored on the memory.
In a sixth aspect, embodiments of the present application provide a machine-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a sub-graph scale prediction method as provided in the first aspect of embodiments of the present application or a distributed training method as provided in the second aspect of embodiments of the present application.
In a seventh aspect, embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the sub-graph scale prediction method as provided in the first aspect of embodiments of the present application or the distributed training method as provided in the second aspect of embodiments of the present application.
According to the sub-graph scale prediction and distributed training method, the sub-graph scale prediction and distributed training device and the electronic equipment, the average node number of the sub-graph is obtained by obtaining the node number, the average degree and the sub-graph number of the graph and utilizing a prediction model according to the node number, the average degree and the sub-graph number; if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is adjusted, and the step of obtaining the average node number of the sub-graph by utilizing a prediction model is performed according to the node number, the average degree and the number of the sub-graph; and if the number of the sub-graph average nodes does not exceed the upper limit of the node number, determining a sub-graph scale prediction result comprising the current number of the sub-graph and the number of the sub-graph average nodes. The prediction model is established based on a preset graph cutting algorithm and a graph cutting result of the graph with the number of the sub-graphs, and the number of sub-graph average nodes obtained by dividing the graph can be obtained based on the model, that is, the number of sub-graph average nodes after dividing can be predicted before dividing the graph. Further, the obtained average node number of the sub-graph is compared with the upper limit of the node number which can be borne by the computing unit, if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is required to be adjusted, namely the average node number of the sub-graph is correspondingly adjusted, so that the average node number of the sub-graph which is finally adjusted does not exceed the upper limit of the node number which can be borne by the computing unit, the possibility that the computing unit can process the sub-graph is increased, and the computing unit can process the sub-graph normally to a certain extent.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other embodiments may be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a schematic diagram of a prior art framework using graph cut, CPU and GPU hybrid collaboration;
FIG. 2 is a schematic flow chart of a sub-graph scale prediction method according to an embodiment of the present application;
FIG. 3 is a sub-graph of the graph cut of an embodiment of the present application;
FIG. 4 is a flow chart of a distributed training method according to an embodiment of the present application;
FIG. 5 is a flow chart of a distributed training method according to another embodiment of the present application;
fig. 6 is a schematic structural diagram of a sub-graph scale prediction apparatus according to an embodiment of the present application;
FIG. 7 is a schematic structural diagram of a distributed training device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. Based on the embodiments herein, a person of ordinary skill in the art would be able to obtain all other embodiments based on the disclosure herein, which are within the scope of the disclosure herein.
Machine learning techniques are currently used in various industries, such as popular applications, and graph processing algorithms are very popular in enterprise digital transformation applications. Numerous graph processing algorithms are currently emerging, with representative graph processing algorithms including: KGE (Knowledge Graph Embedding ) algorithm and GNN (Graph Neural Network, graph neural network) algorithm.
When the current graph processing algorithm is applied in the floor, two main challenges exist in engineering.
One is how graph processing applications can be developed on a large scale graph. In this regard, the main technical means in the current industry is to use a graph cutting algorithm to complete the cutting of a large-scale graph, and then use a distributed computing unit to process the cut subgraph, so as to solve the problem of large data volume in a data parallel manner. The other is how high-performance completion graph processing applications. In this regard, the main technical means in the current industry is to execute a graphics processing algorithm using a high-performance computing unit typified by a GPU (Graphics Processing Unit ), or to execute the graphics processing algorithm in cooperation with a GPU using a hybrid including a CPU (Central Processing Unit ).
As shown in fig. 1, a graph cut, CPU and GPU hybrid synergistic approach is used to achieve high performance to accomplish large scale graph processing. In fact, there are still some real problems when the graph process is truly floor-standing. Factors such as different graph scales and different graph structures, the number of the sub-graphs after graph cutting and the like have great influence on the sub-graph scales after graph cutting. For example, when the degree of the graph is relatively large and the number of sub-graphs after the graph is cut is not large, the number of nodes in the sub-graphs after the graph is cut is very easy to approach the number of nodes in the whole graph, so that the performance improvement effect of the distributed driving mode is limited.
In order to address the above issues, prior to the application of a subgraph, an effort was made to solve how many subgraphs should be cut to enable the distributed data to complete real performance acceleration in parallel, and an effort was made to solve how to predict the required computing resources. However, most of the current approaches are: and manually completing the graph cutting experiment by field coding according to the actual scale of the graph and counting the scale of the subgraph. The method requires a complex manual coding process and continuous experiments, has huge workload, and is difficult to meet the demand of prediction calculation resources; particularly under the condition that cloud environment resources are elastically extensible, the manual mode cannot be intelligently adapted.
In order to ensure that a computing unit can normally process a sub-graph to a certain extent and realize prediction of computing resources required by sub-graph processing, the embodiment of the application provides a sub-graph scale prediction and distributed training method, device and electronic equipment.
The terms in the embodiments of the present application are explained as follows:
linear regression: a statistical analysis method for determining the quantitative relationship of interdependence between two or more variables by regression analysis in mathematical statistics.
Subgraph: in the graph theory, the finger node set and the edge set are graphs of a subset of the node set and a subset of the edge set of a certain graph, respectively.
Graph cutting: the concept in graph theory is to cut a graph into a plurality of subgraphs based on a graph cutting algorithm according to nodes or edges, and the graph cutting does not affect the original topological relation between the nodes and the edges in the subgraphs. The graph cut algorithm may be different depending on the requirements.
Degree: the concept in graph theory, the degree of a node refers to the number of edges of the node.
KGE: map the graph data into a low-dimensional vector that reflects the topology of the graph, on which the existing machine learning algorithm can be used directly in subsequent applications.
GNN: in the method for processing the graph information based on deep learning on the graph data, compared with a fully-connected neural network connection layer, the graph neural network has more adjacent matrix concepts and reflects the relation between nodes on the graph. The graph neural network can output vector representations of all nodes finally and is used for tasks such as subsequent node classification, link prediction and the like.
Expansion rate: when scanning adjacent neighbors based on a node set in a certain sub-graph, the ratio of the number of the scanned neighbor nodes which have no intersection with the node set of the sub-graph to the number of all the scanned neighbor nodes reflects the expansion degree of the new sub-graph scale formed after scanning the neighbors compared with the original sub-graph scale.
Next, first, a sub-graph scale prediction method provided in the embodiment of the present application will be described.
The execution body of the sub-graph scale prediction method provided by the embodiment of the application may be an electronic device with specific core processing capability, for example, a management node in a distributed system, or a data processing server. The mode of implementing the sub-graph scale prediction method provided by the embodiment of the present application may be at least one of software, a hardware circuit and a logic circuit disposed in the execution body.
As shown in fig. 2, the sub-graph scale prediction method provided in the embodiment of the present application may include the following steps.
S201, the number of nodes, the average degree and the number of subgraphs of the graph are obtained.
Wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is the average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of sub-graphs is initially set.
S202, obtaining the average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number.
The prediction model is established based on a preset graph cutting algorithm and a graph of the subgraph number to obtain a cutting result of the subgraph.
S203, judging whether the average node number of the subgraph exceeds the upper limit of the node number which can be borne by the computing unit, if so, executing S204, and if not, executing S205.
S204, adjusting the number of sub-graphs. Execution returns to S202.
S205, determining a sub-graph scale prediction result, wherein the sub-graph scale prediction result comprises the current number of sub-graphs and the number of sub-graph average nodes.
By applying the embodiment of the application, the average node number of the subgraph is obtained by obtaining the node number, the average degree and the subgraph number of the graph and utilizing a prediction model according to the node number, the average degree and the subgraph number; if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is adjusted, and the step of obtaining the average node number of the sub-graph by utilizing a prediction model is performed according to the node number, the average degree and the number of the sub-graph; and if the number of the sub-graph average nodes does not exceed the upper limit of the node number, determining a sub-graph scale prediction result comprising the current number of the sub-graph and the number of the sub-graph average nodes. The prediction model is established based on a preset graph cutting algorithm and a graph cutting result of the graph with the number of the sub-graphs, and the number of sub-graph average nodes obtained by dividing the graph can be obtained based on the model, that is, the number of sub-graph average nodes after dividing can be predicted before dividing the graph. Further, the obtained average node number of the sub-graph is compared with the upper limit of the node number which can be borne by the computing unit, if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is required to be adjusted, namely the average node number of the sub-graph is correspondingly adjusted, so that the average node number of the sub-graph which is finally adjusted does not exceed the upper limit of the node number which can be borne by the computing unit, the possibility that the computing unit can process the sub-graph is increased, and the computing unit can process the sub-graph normally to a certain extent.
After obtaining a graph including a plurality of nodes, the number of nodes of the graph may be obtained by means of node statistics. The degree of a node represents the number of edges of the node, specifically, the number of edges of the node can be directly set as the degree of the node, or the degree of the node can be set in a mode of distributing weights, and the larger the weight is, the more the number of edges of the node is. The degree of each node can be obtained by counting the number of edges of each node, and the average degree of the graph can be obtained by calculating the average value of the degrees of all the nodes in the graph, wherein the average degree of the graph represents the association condition among the nodes in the graph. The larger the average degree is, the more complex the association relation of the nodes in the explanatory diagram is. Of course, the average degree of the graph may be calculated by the user or the front-end device when generating the graph, and the electronic device of the present application is informed in the form of graph information, so that the electronic device may obtain the average degree of the graph from the graph information while obtaining the graph. In the initial case, the number of sub-images is a set value input by the user, that is to say, how many sub-images the image is desired to be divided into, and the value can also inform the electronic device in the form of image information when the image is generated.
After the number, the average degree and the number of the sub-images of the nodes of the image are obtained, the average node number of the segmented sub-images is predicted by using a prediction model, specifically, the prediction model is established based on a preset image cutting algorithm and a cutting result of the sub-images obtained by cutting the image by the number of the sub-images, the number, the average degree and the number of the sub-images jointly determine the sizes of the sub-images cut by the preset image cutting algorithm, so that the prediction model is established based on the experience, and further, the prediction result of the average node of the sub-images can be obtained by using the prediction model.
In an implementation manner of the embodiment of the present application, the step of cutting the graph based on a preset graph cutting algorithm and the number of sub-graphs to obtain the sub-graphs may specifically be: selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
According to the preset graph cutting algorithm adopted by the embodiment of the application, multiple groups of central nodes are selected from all nodes according to the number of the subgraphs, for example, if the number of the subgraphs is 10, 10 groups of central nodes are selected from all the nodes, and the selected strategy is to select root nodes as much as possible or select according to the principle of uniform distribution. Starting from the center node, the process can outwards continue to expand the multi-order neighbor nodes and edges according to the needs based on the outwards expanded scanning of the neighbor relation of the center node, and finally obtaining a plurality of segmented subgraphs according to the obtained neighbor nodes and edges.
Taking a second-order neighbor node as an example, for any group of center nodes, based on the neighbor relation of the group of center nodes, expanding and scanning outwards to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges, wherein the steps specifically can be as follows: aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
As shown in fig. 3, the central node is node 1, 2, 3, based on the neighbor relation of the central node, the outward expansion scan identifies that node 2, 3, 4, 6 is the first-order neighbor node of node 1, 5, 7 is the first-order neighbor node of node 2, node 1, 6, 8, 9 is the first-order neighbor node of node 3, it can be determined that node 1, 2, 3, 4, 5, 6, 7, 8, 9 and the edges between these nodes form a sub-graph (as the smaller dashed circle in fig. 3), and also identifies that node 3, 5, 6, 7, 8, 9, 10, 11, 12, 15 is the second-order neighbor node of node 1, node 3, 6, 4, 13, 14 is the second-order neighbor node of node 2, node 1, 2, 4, 6, 15, 16, 17, 18 is the second-order neighbor node of node 3, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 and the edges between these nodes form a larger dashed circle (as the middle dashed circle in fig. 3).
In one implementation manner of the embodiment of the present application, the prediction model is specifically shown in formula (1):
Figure BDA0002995460580000121
wherein V is S (A) In order to obtain the average node number of the subgraphs after cutting the graph A, V (A) is the node number of the graph A, C (A) is the subgraph number of the graph A, d (A) is the average degree of the graph A, and gamma 1 For outwards expanding and scanning expansion rate of first-order neighbor nodes based on neighbor relation of center node, gamma 2 And expanding and scanning the expansion rate of the second-order neighbor node for outward expansion based on the neighbor relation of the first-order neighbor node.
As can be seen from fig. 3, each time a neighbor node is scanned based on a sub-graph node set (e.g., a set of center nodes, including a set of center nodes and first-order neighbor nodes, etc.) being expanded outwards to construct a new sub-graph, the scanned neighbor node may be either an existing node in the sub-graph node set or a newly brought node. These newly brought nodes are introduced as sub-graphs expanding outwards, for which the expansion ratio is defined:
expanding and scanning the expansion rate of the first-order neighbor nodes outwards based on the neighbor relation of the center node:
Figure BDA0002995460580000122
expanding and scanning the expansion rate of the second-order neighbor node outwards based on the neighbor relation of the first-order neighbor node:
Figure BDA0002995460580000123
the expansion rate shows the size of the current subgraph's ability to introduce new nodes. The larger the expansion ratio is, the more the number of new nodes introduced by the neighbor of the expansion scanning of the current subgraph is; conversely, the smaller the expansion ratio is, the smaller the number of new nodes introduced by the neighbor of the expansion scanning of the current subgraph is.
Taking fig. 3 as an example, the expansion ratio of the first-order neighbor node is expanded and scanned outwards based on the neighbor relation of the center node is as follows: gamma ray 1 =6/11 expansion ratio of the outwardly expanding scan second order neighbor node based on the neighbor relation of the first order neighbor node: gamma ray 2 ＝9/23。
Based on the above, a predictive model may be built, taking graph a as an example:
the number of starting nodes of the subgraph is as follows:
Figure BDA0002995460580000131
the number of newly introduced first-order neighbor nodes is as follows:
Figure BDA0002995460580000132
/>
the number of newly introduced second-order neighbor nodes is as follows:
Figure BDA0002995460580000133
and (3) synthesizing to obtain the prediction model shown in the formula (1).
Based on analysis of the graph, there are two rules related to graph cut:
assuming that two graphs are provided, namely a graph A and a graph B, the number of nodes is respectively marked as V (A) and V (B), the average degree is respectively marked as d (A) and d (B), the number of sub-graphs is respectively marked as C (A) and C (B), and the average number of sub-graphs after graph cutting is marked as V S (A) And V S (B)。
First law:
V(A)<v (B), d (a) =d (B), C (a) =c (B),
Figure BDA0002995460580000134
the reason for this result is that when V (B) is larger, the data of the initial center node of the sub-graph after graph cutting is relatively more, the possibility of introducing new nodes into the outward expansion neighbor will be reduced, the expansion rate of introducing new nodes will slide down, and the result is finally reflected in V S (B) Number of digitsThe value is smaller.
Second law:
V(A)＝V(B)，d(A)＝d(B)，C(A)<in the case of the component C (B),
Figure BDA0002995460580000135
the reason for this result is that when C (A) is smaller, the data of the initial center node of the sub-graph cut by the graph is relatively more, the possibility of introducing new nodes into the outward expansion neighbor is reduced, the expansion rate of introducing the new nodes is reduced, and the expansion rate is finally reflected in V S (A) The numerical value is smaller.
Based on the two rules, it can be known that the expansion rate has strong correlation with the number of nodes and the average degree of the graph, so in one embodiment of the present application, a linear regression model may be established based on the strong correlation, where the linear regression model is used to fit the expansion rate.
Specifically, the linear regression model may be as shown in equation (2):
Figure BDA0002995460580000141
wherein omega 1 、ω 2 、ω 3 、ω 4 、ω 5 、ω 6 Is a preset weight; b is a constant; alpha is the relative expansion rate gamma 1 And gamma 2 Performing fine tuning.
In the embodiment of the application, the prediction model is utilized, after the node number, the average degree and the sub-graph number of the graph are given, the average node number of the sub-graph can be effectively predicted, and as the node number which can bear and run is limited, the cost of repeated error trial and error on the existing resources is reduced, and meanwhile, the number of resources to be expanded can be predicted according to the requirement, for example, the number of new calculation units is increased, so that the rapid landing of the graph processing algorithm in a real scene is accelerated.
The embodiment of the application also provides a distributed training method. The execution subject of the distributed training method provided by the embodiment of the application may be an electronic device with a data processing function, such as a model training server, a data analysis machine, and the like. The distributed training method provided in the embodiments of the present application may be at least one of software, a hardware circuit, and a logic circuit disposed in the execution body.
As shown in fig. 4, a distributed training method provided in an embodiment of the present application may include the following steps.
S401, cutting the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain each subgraph.
Wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents the association of the two nodes; the number of sub-graphs is obtained based on the sub-graph scale prediction method of the embodiment shown in fig. 2.
S402, each sub-graph is sent to each computing unit respectively, so that the preset graph processing algorithm is trained by each sub-graph in each computing unit respectively.
By applying the embodiment of the application, because the number of the subgraphs is obtained based on the subgraph scale prediction method of the embodiment shown in fig. 2, the average node number of the subgraphs obtained by cutting the subgraphs among the graphs based on the preset graph cutting algorithm and the subgraph number does not exceed the upper limit of the node number which can be borne by the computing unit, when the distributed training is carried out, the possibility that the computing unit can process the subgraphs is increased, so that the normal running of the distributed training is ensured to a certain extent, and the training efficiency is improved.
The specific distributed training process is shown in fig. 5, and mainly comprises three steps: splitting the nodes to obtain a plurality of groups of central nodes; cutting the graph by using a preset graph cutting algorithm based on each group of central nodes to obtain each sub-graph; and loading each sub-graph serving as an independent data partition to an independent computing unit, and training a sub-graph processing algorithm in each computing unit.
In one embodiment of the present application, S401 may specifically be: selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
According to the preset graph cutting algorithm adopted by the embodiment of the application, multiple groups of central nodes are selected from all nodes according to the number of the subgraphs, for example, if the number of the subgraphs is 10, 10 groups of central nodes are selected from all the nodes, and the selected strategy is to select root nodes as much as possible or select according to the principle of uniform distribution. Starting from the center node, the process can outwards continue to expand the multi-order neighbor nodes and edges according to the needs based on the outwards expanded scanning of the neighbor relation of the center node, and finally obtaining a plurality of segmented subgraphs according to the obtained neighbor nodes and edges.
Taking a second-order neighbor node as an example, for any group of center nodes, based on the neighbor relation of the group of center nodes, expanding and scanning outwards to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges, wherein the steps specifically can be as follows: aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
As shown in fig. 3, the central node is node 1, 2, 3, based on the neighbor relation of the central node, the outward expansion scan identifies that node 2, 3, 4, 6 is the first-order neighbor node of node 1, 5, 7 is the first-order neighbor node of node 2, node 1, 6, 8, 9 is the first-order neighbor node of node 3, it can be determined that node 1, 2, 3, 4, 5, 6, 7, 8, 9 and the edges between these nodes form a sub-graph (as the smaller dashed circle in fig. 3), and also identifies that node 3, 5, 6, 7, 8, 9, 10, 11, 12, 15 is the second-order neighbor node of node 1, node 3, 6, 4, 13, 14 is the second-order neighbor node of node 2, node 1, 2, 4, 6, 15, 16, 17, 18 is the second-order neighbor node of node 3, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 and the edges between these nodes form a larger dashed circle (as the middle dashed circle in fig. 3).
After each sub-graph is obtained through cutting, each sub-graph is used as an independent data partition, and is loaded to an independent computing unit such as a server, a CPU, a GPU and the like, and a preset graph processing algorithm (such as a KGE algorithm model, a GNN model and the like) is trained in each computing unit.
The specific training process can be as follows: calculating the subgraph of each calculation unit by using a graph processing algorithm, carrying out loss calculation on the subgraph of each calculation unit based on a calculation result to obtain the gradient of the graph processing algorithm in the calculation unit, converging the graph processing algorithm among the gradients of each calculation unit, carrying out calculation such as averaging on the converged gradient, and the like, sending the calculated gradient to each calculation unit, regulating the parameters of the graph processing algorithm by each calculation unit based on the calculated gradient, and returning to execute the step of calculating the subgraph of the calculation unit by using the graph processing algorithm, completing gradient convergence and distribution among each calculation unit, starting the next iteration until the gradient of each node converges or the designated iteration number is reached, and determining that training is completed.
Of course, each sub-graph can be divided during training to obtain a training set, an effective set and a test set, and training and evaluation of the graph processing algorithm are carried out in such a way that node data in the test set is input into the trained algorithm to test the operation effect of the algorithm.
Corresponding to the method embodiment shown in fig. 2, the embodiment of the present application provides a sub-map scale prediction apparatus, as shown in fig. 6, which may include:
an obtaining module 610, configured to obtain the number of nodes, the average degree, and the number of sub-graphs of the graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is the average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of the subgraphs is a set value under the initial condition;
the prediction module 620 is configured to obtain an average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number; the prediction model is established based on a preset graph cutting algorithm and a graph of the subgraph number to obtain a cutting result of the subgraph;
an adjustment module 630, configured to adjust the number of sub-graphs if the average number of node points of the sub-graphs exceeds the upper limit of the number of node points that can be carried by the calculation unit;
and the determining module 640 is configured to determine a sub-graph scale prediction result if the average node number of the sub-graphs does not exceed the node number upper limit, where the sub-graph scale prediction result includes the current sub-graph number and the average node number of the sub-graphs.
Optionally, the apparatus further comprises:
the prediction model building module is used for selecting a plurality of groups of center nodes from all the nodes according to the number of the subgraphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, the prediction model building module is specifically configured to, for any group of center nodes, based on the neighbor relation of the group of center nodes, perform outward expansion scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determine that the group of center nodes, the first-order neighbor nodes, and edges between the group of center nodes and the first-order neighbor nodes form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
Optionally, the prediction model is:
Figure BDA0002995460580000171
wherein V is S (A) In order to obtain the average node number of the subgraphs after cutting the graph A, V (A) is the node number of the graph A, C (A) is the subgraph number of the graph A, d (A) is the average degree of the graph A, and gamma 1 For outwards expanding and scanning expansion rate of first-order neighbor nodes based on neighbor relation of center node, gamma 2 And expanding and scanning the expansion rate of the second-order neighbor node for outward expansion based on the neighbor relation of the first-order neighbor node.
Alternatively, the expansion rate gamma 1 And gamma 2 The linear regression model is obtained through fitting, and is established based on the relevance of the expansion rate, the number of nodes and the average degree.
Alternatively, the linear regression model is:
Figure BDA0002995460580000172
γ 2 ＝α*γ 1
wherein omega 1 、ω 2 、ω 3 、ω 4 、ω 5 、ω 6 Is a preset weight; b is a constant; alpha is the relative expansion rate gamma 1 And gamma 2 Performing fine tuning.
By applying the embodiment of the application, the average node number of the subgraph is obtained by obtaining the node number, the average degree and the subgraph number of the graph and utilizing a prediction model according to the node number, the average degree and the subgraph number; if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is adjusted, and the step of obtaining the average node number of the sub-graph by utilizing a prediction model is performed according to the node number, the average degree and the number of the sub-graph; and if the number of the sub-graph average nodes does not exceed the upper limit of the node number, determining a sub-graph scale prediction result comprising the current number of the sub-graph and the number of the sub-graph average nodes. The prediction model is established based on a preset graph cutting algorithm and a graph cutting result of the graph with the number of the sub-graphs, and the number of sub-graph average nodes obtained by dividing the graph can be obtained based on the model, that is, the number of sub-graph average nodes after dividing can be predicted before dividing the graph. Further, the obtained average node number of the sub-graph is compared with the upper limit of the node number which can be borne by the computing unit, if the average node number of the sub-graph exceeds the upper limit of the node number which can be borne by the computing unit, the number of the sub-graph is required to be adjusted, namely the average node number of the sub-graph is correspondingly adjusted, so that the average node number of the sub-graph which is finally adjusted does not exceed the upper limit of the node number which can be borne by the computing unit, the possibility that the computing unit can process the sub-graph is increased, and the computing unit can process the sub-graph normally to a certain extent.
Corresponding to the method embodiment shown in fig. 4, the embodiment of the present application provides a distributed training device, as shown in fig. 7, which may include:
the cutting module 710 is configured to cut the graph based on a preset graph cutting algorithm and the number of sub-graphs, so as to obtain each sub-graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the number of the subgraphs is obtained based on the subgraph scale prediction method;
and the sending module 720 is configured to send each sub-graph to each computing unit, so as to train the preset graph processing algorithm by using each sub-graph in each computing unit.
Optionally, the cutting module 710 may be specifically configured to select multiple groups of central nodes from the nodes according to the number of sub-graphs; and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
Optionally, the cutting module 710 may be specifically configured to, for any group of center nodes, perform outward expansion scanning based on a neighbor relation of the group of center nodes to obtain first-order neighbor nodes and edges of the group of center nodes, and determine that the group of center nodes, the first-order neighbor nodes, and edges between the group of center nodes and the first-order neighbor nodes form a subgraph; based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
By applying the embodiment of the application, because the number of the subgraphs is obtained based on the subgraph scale prediction method, the average node number of the subgraphs obtained by cutting the subgraphs between the graphs based on the preset graph cutting algorithm and the subgraph number does not exceed the upper limit of the node number which can be borne by the computing unit, when the distributed training is carried out, the possibility that the computing unit can process the subgraphs is increased, so that the normal running of the distributed training is ensured to a certain extent, and the training efficiency is improved.
The embodiment of the application also provides an electronic device, as shown in fig. 8, including a processor 801 and a memory 802; wherein the memory 802 is used for storing a computer program; the processor 801 is configured to implement the sub-picture scale prediction method or the distributed training method when executing the computer program stored in the memory 802.
The Memory may include RAM (Random Access Memory ) or NVM (Non-volatile Memory), such as at least one magnetic disk Memory. Optionally, the memory may be at least one memory device located remotely from the processor.
The processor may be a general-purpose processor, including a CPU (Central Processing Unit ), NP (Network Processor, network processor), etc.; but also DSP (Digital Signal Processing, digital signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field-Programmable Gate Array, field programmable gate array), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
In addition, the embodiment of the application also provides a machine-readable storage medium, and a computer program is stored in the machine-readable storage medium, and when the computer program is executed by a processor, the sub-graph scale prediction method or the distributed training method is realized.
Embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the above sub-graph scale prediction method or the distributed training method.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, DSL (Digital Subscriber Line, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD (Digital Versatile Disc, digital versatile Disk)), or a semiconductor medium (e.g., an SSD (Solid State Disk)), or the like.
For sub-picture scale prediction apparatus, distributed training apparatus, electronic device, machine readable storage medium and computer program product embodiments, the description is relatively simple as it relates to method content substantially similar to the method embodiments described above, and reference is made to the description of the method embodiments in part.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for sub-picture scale prediction apparatus, distributed training apparatus, electronic device, machine-readable storage medium and computer program product embodiments, the description is relatively simple as it is substantially similar to method embodiments, as relevant to see the partial description of method embodiments.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims (13)

1. A sub-graph scale prediction method, the method comprising:
acquiring the number, average degree and number of sub-graphs of nodes of the graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is an average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of the subgraphs is a set value in the initial condition;
Obtaining the average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number; the prediction model cuts the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain a cutting result of the subgraphs;
if the average node number of the subgraph exceeds the upper limit of the node number which can be borne by the computing unit, the subgraph number is adjusted, and the step of obtaining the average node number of the subgraph by utilizing a prediction model according to the node number, the average degree and the subgraph number is executed in a returning mode;
and if the number of the sub-graph average nodes does not exceed the upper limit of the number of the nodes, determining a sub-graph scale prediction result, wherein the sub-graph scale prediction result comprises the current number of the sub-graphs and the number of the sub-graph average nodes.
2. The method according to claim 1, wherein the step of cutting the graph based on a preset graph cutting algorithm and the number of sub-graphs to obtain sub-graphs comprises:
selecting a plurality of groups of center nodes from the nodes according to the number of the subgraphs;
and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
3. The method according to claim 2, wherein the step of expanding the scan outwards for any group of center nodes based on the neighbor relation of the group of center nodes to obtain multi-level neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to each obtained level of neighbor nodes and edges comprises:
aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph;
based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
4. A method according to claim 3, wherein the predictive model is:
Figure FDA0002995460570000021
wherein V is S (A) In order to obtain the average node number of the subgraphs after cutting the graph A, V (A) is the node number of the graph A, C (A) is the subgraph number of the graph A, d (A) is the average degree of the graph A, and gamma 1 For outwards expanding and scanning expansion rate of first-order neighbor nodes based on neighbor relation of central node, gamma 2 And expanding and scanning the expansion rate of the second-order neighbor node outwards based on the neighbor relation of the first-order neighbor node.
5. The method of claim 4, wherein the expansion rate γ 1 And gamma 2 And fitting by a linear regression model, wherein the linear regression model is established based on the relevance of the expansion rate, the number of the nodes and the average degree.
6. The method of claim 5, wherein the linear regression model is:
Figure FDA0002995460570000022
γ 2 ＝α*γ 1
wherein omega 1 、ω 2 、ω 3 、ω 4 、ω 5 、ω 6 Is a preset weight; b is a constant; alpha is the expansion rate gamma 1 And gamma 2 Performing fine tuning.
7. A distributed training method, the method comprising:
cutting the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain each subgraph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the number of sub-figures being based on the method according to any of claims 1-6;
and respectively sending the subgraphs to each computing unit so as to train a preset graph processing algorithm by utilizing the subgraphs in each computing unit.
8. The method according to claim 7, wherein the step of cutting the graph to obtain each sub-graph based on a preset graph cutting algorithm and the number of sub-graphs includes:
selecting a plurality of groups of center nodes from the nodes according to the number of the subgraphs;
and aiming at any group of center nodes, outwards expanding and scanning based on the neighbor relation of the group of center nodes to obtain multi-order neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to the obtained multi-order neighbor nodes and edges.
9. The method according to claim 8, wherein the step of expanding the scan outwards for any group of center nodes based on the neighbor relation of the group of center nodes to obtain multi-level neighbor nodes and edges of the group of center nodes, and obtaining a plurality of subgraphs according to each obtained level of neighbor nodes and edges comprises:
aiming at any group of center nodes, based on the neighbor relation of the group of center nodes, outwards expanding and scanning to obtain first-order neighbor nodes and edges of the group of center nodes, and determining the group of center nodes, the first-order neighbor nodes and edges among the group of center nodes and the first-order neighbor nodes to form a subgraph;
based on the neighbor relation of the first-order neighbor nodes, outward expansion scanning is performed to obtain second-order neighbor nodes and edges of the group of center nodes, and the center nodes, the first-order neighbor nodes, the second-order neighbor nodes, edges between the center nodes and the first-order neighbor nodes and edges between the first-order neighbor nodes and the second-order neighbor nodes are determined to form a subgraph.
10. A sub-map scale prediction apparatus, the apparatus comprising:
the acquisition module is used for acquiring the number, average degree and sub-graph number of the nodes of the graph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the average degree is an average value of the degrees of all nodes in the graph, and the degrees of the nodes represent the number of edges of the nodes; the number of the subgraphs is a set value in the initial condition;
the prediction module is used for obtaining the average node number of the subgraph by using a prediction model according to the node number, the average degree and the subgraph number; the prediction model cuts the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain a cutting result of the subgraphs;
the adjusting module is used for adjusting the number of the subgraph if the average node number of the subgraph exceeds the upper limit of the node number which can be borne by the computing unit;
and the determining module is used for determining a sub-graph scale prediction result if the number of the sub-graph average nodes does not exceed the upper limit of the node number, wherein the sub-graph scale prediction result comprises the current number of sub-graphs and the number of the sub-graph average nodes.
11. A distributed training apparatus, the apparatus comprising:
the cutting module is used for cutting the graph based on a preset graph cutting algorithm and the number of the subgraphs to obtain each subgraph; wherein the graph comprises a plurality of nodes; the connection line between two nodes forms an edge, and represents that the two nodes are related; the number of sub-figures being based on the method according to any of claims 1-6;
and the sending module is used for respectively sending the subgraphs to the computing units so as to train a preset graph processing algorithm by utilizing the subgraphs in the computing units.
12. An electronic device comprising a processor and a memory;
the memory is used for storing a computer program;
the processor being adapted to implement the method of any of claims 1-6 or the method of any of claims 7-9 when executing a computer program stored on the memory.
13. A machine-readable storage medium, characterized in that it has stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1-6 or the method of any one of claims 7-9.
CN202110328443.3A 2021-03-26 2021-03-26 Sub-graph scale prediction and distributed training method and device and electronic equipment Active CN112990332B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110328443.3A CN112990332B (en) 2021-03-26 2021-03-26 Sub-graph scale prediction and distributed training method and device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110328443.3A CN112990332B (en) 2021-03-26 2021-03-26 Sub-graph scale prediction and distributed training method and device and electronic equipment

Publications (2)

Publication Number Publication Date
CN112990332A CN112990332A (en) 2021-06-18
CN112990332B true CN112990332B (en) 2023-06-02

Family

ID=76333942

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110328443.3A Active CN112990332B (en) 2021-03-26 2021-03-26 Sub-graph scale prediction and distributed training method and device and electronic equipment

Country Status (1)

Country Link
CN (1) CN112990332B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102073708A (en) * 2010-12-30 2011-05-25 东北大学 Large-scale uncertain graph database-oriented subgraph query method
CN103645952A (en) * 2013-08-08 2014-03-19 中国人民解放军国防科学技术大学 Non-accurate task parallel processing method based on MapReduce
CN105426764A (en) * 2015-11-16 2016-03-23 北京航空航天大学 Submodel optimization based parallel abnormal sub-graph detection method and system
CN106528755A (en) * 2016-10-28 2017-03-22 东软集团股份有限公司 Hot topic generation method and device
CN108681482A (en) * 2018-05-16 2018-10-19 腾讯科技（深圳）有限公司 Method for scheduling task based on diagram data and device
CN109952742A (en) * 2018-12-04 2019-06-28 区链通网络有限公司 Graph structure processing method, system, the network equipment and storage medium
CN111382320A (en) * 2020-03-18 2020-07-07 军事科学院系统工程研究院系统总体研究所 Large-scale data increment processing method for knowledge graph
CN112100450A (en) * 2020-09-07 2020-12-18 厦门渊亭信息科技有限公司 Graph calculation data segmentation method, terminal device and storage medium

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10198654B2 (en) * 2016-06-24 2019-02-05 Ebay Inc. Automated saliency map estimation
US10885118B2 (en) * 2017-05-12 2021-01-05 Futurewei Technologies, Inc. Incremental graph computations for querying large graphs

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102073708A (en) * 2010-12-30 2011-05-25 东北大学 Large-scale uncertain graph database-oriented subgraph query method
CN103645952A (en) * 2013-08-08 2014-03-19 中国人民解放军国防科学技术大学 Non-accurate task parallel processing method based on MapReduce
CN105426764A (en) * 2015-11-16 2016-03-23 北京航空航天大学 Submodel optimization based parallel abnormal sub-graph detection method and system
CN106528755A (en) * 2016-10-28 2017-03-22 东软集团股份有限公司 Hot topic generation method and device
CN108681482A (en) * 2018-05-16 2018-10-19 腾讯科技（深圳）有限公司 Method for scheduling task based on diagram data and device
CN109952742A (en) * 2018-12-04 2019-06-28 区链通网络有限公司 Graph structure processing method, system, the network equipment and storage medium
CN111382320A (en) * 2020-03-18 2020-07-07 军事科学院系统工程研究院系统总体研究所 Large-scale data increment processing method for knowledge graph
CN112100450A (en) * 2020-09-07 2020-12-18 厦门渊亭信息科技有限公司 Graph calculation data segmentation method, terminal device and storage medium

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Graph Partitioning for Distributed Graph Processing;Makoto Onizuka 等;《10.1007/s41019-017-0034-4》;全文 *
Parallelizing Maximal Clique Enumeration Over Graph Data;Qun Chen 等;《Springer》;全文 *
基于解耦概要图的大规模图数据高效分布式挖掘算法;李玲 等;《计算机学报》;全文 *

Also Published As

Publication number Publication date
CN112990332A (en) 2021-06-18

Similar Documents

Publication Publication Date Title
US11018979B2 (en) System and method for network slicing for service-oriented networks
CN109993299B (en) Data training method and device, storage medium and electronic device
CN113261003A (en) Generating integrated circuit floor plan using neural network
CN108667657B (en) SDN-oriented virtual network mapping method based on local feature information
CN111406264A (en) Neural architecture search
CN112073452B (en) Mobile edge computing task allocation method with effective energy and limited resources
Chen et al. $ d $ d-Simplexed: Adaptive Delaunay Triangulation for Performance Modeling and Prediction on Big Data Analytics
US20210209481A1 (en) Methods and systems for dynamic service performance prediction using transfer learning
CN112436992B (en) Virtual network mapping method and device based on graph convolution network
CN113746798B (en) Cloud network shared resource abnormal root cause positioning method based on multi-dimensional analysis
CN113821332B (en) Method, device, equipment and medium for optimizing efficiency of automatic machine learning system
CN112732444A (en) Distributed machine learning-oriented data partitioning method
Badri et al. A sample average approximation-based parallel algorithm for application placement in edge computing systems
CN113886092A (en) Computation graph execution method and device and related equipment
CN112990332B (en) Sub-graph scale prediction and distributed training method and device and electronic equipment
CN111124439B (en) Intelligent dynamic unloading algorithm with cloud edge cooperation
Huang et al. No-wait two-stage multiprocessor flow shop scheduling with unit setup
Liu Performance Test and Improvement of Computer Network Virtualization Software in Cloud Computing Environment
TW202145078A (en) Computing method with dynamic minibatch sizes and computing system and computer-readable storage media for performing the same
Ismaeel et al. A systematic cloud workload clustering technique in large scale data centers
WO2023029155A1 (en) Spark graphx parameter tuning method and apparatus, electronic device and storage medium
Rigoni et al. Metamodels for fast multi-objective optimization: trading off global exploration and local exploitation
Ovalle et al. Distributed Cache Strategies for Machine Learning Classification Tasks over Cluster Computing Resources
CN112948087A (en) Task scheduling method and system based on topological sorting
CN113343040A (en) Automatic incremental method, device, equipment and storage medium for graph algorithm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant