CN114511100A - Graph model task implementation method and system supporting multi-engine framework - Google Patents
Graph model task implementation method and system supporting multi-engine framework Download PDFInfo
- Publication number
- CN114511100A CN114511100A CN202210393596.0A CN202210393596A CN114511100A CN 114511100 A CN114511100 A CN 114511100A CN 202210393596 A CN202210393596 A CN 202210393596A CN 114511100 A CN114511100 A CN 114511100A
- Authority
- CN
- China
- Prior art keywords
- graph
- machine learning
- target
- engine
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the specification provides a method and a system for realizing graph model tasks supporting a multi-engine framework, and the method and the system comprise a graph model task processing method and a graph model task deployment method, wherein the task processing method comprises the following steps: acquiring graph data in a preset data format; determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames; converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning frame to obtain target input data; and providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on the machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
Description
Technical Field
The application relates to the technical field of machine learning, in particular to a graph model task implementation method and system.
Background
With the development of computer technology, machine learning related technologies are applied to various fields and play an important role in data processing or data analysis in various fields. The machine learning engine can provide a computing structure component of an extensible machine learning field and can implement various data processing tasks of the machine learning field. For example, in the related technical field of map data such as a knowledge graph, data processing tasks such as model training, model prediction and the like of a map model can be realized through a machine learning engine; for example, a graph learning framework may be constructed based on a machine learning engine, and data processing tasks such as model training and model prediction of a graph model may be realized by the graph learning framework.
The machine learning engine includes a plurality of types (e.g., Tensorflow, Pyorch, MxNet, etc.), each of which may also support the creation of one or more types of image learning frameworks. Different machine learning engines and graph learning frameworks have respective characteristics and advantages, but the respective input data formats, model codes, model training, task deployment modes and the like are greatly different, so that great complexity is brought to business application.
Therefore, a method and a system for implementing graph model tasks supporting multiple engine frameworks are needed to reduce the complexity of implementing graph model tasks supporting multiple engine frameworks.
Disclosure of Invention
One aspect of the present specification provides a graph model task processing method supporting a multi-engine framework, which is used for graph model task processing, and the method includes: acquiring graph data in a preset data format; determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames; converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning frame to obtain target input data; and providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on the machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
Another aspect of the specification provides a graph model task processing system supporting a multi-engine framework, the system comprising: the first data acquisition module is used for acquiring graph data in a preset data format; the target determining module is used for determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames; the target input data acquisition module is used for converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning frame to obtain target input data; and the task realization module is used for providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on a machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
Another aspect of the present specification provides a graph model task processing apparatus supporting a multi-engine framework, comprising at least one storage medium and at least one processor, the at least one storage medium storing computer instructions; the at least one processor is configured to execute the computer instructions to implement a graph model task implementation method supporting a multi-engine framework as described above.
Another aspect of the present specification provides another graph model task deployment method supporting a multi-engine framework, for graph model task deployment, including: acquiring graph data in a preset data format and a machine learning model supported by a target machine learning engine or a target graph learning framework; receiving task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames and appointing a task deployment mode; and submitting graph data in a preset data format and the machine learning model to one or more computing nodes based on a task deployment mode so that the computing nodes realize graph model tasks based on the graph model task implementation method supporting the multi-engine framework.
Another aspect of the present specification provides another graph model task deployment system supporting a multi-engine framework, the system comprising: the second data acquisition module is used for acquiring graph data in a preset data format and a machine learning model supported by a target machine learning engine or a target graph learning framework; the configuration information receiving module is used for receiving task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames and appointing a task deployment mode; and the task submitting module is used for submitting the graph data in a preset data format and the machine learning model to one or more computing nodes based on the task deployment mode so that the computing nodes can realize the graph model task based on the graph model task realization method supporting the multi-engine framework.
Another aspect of the present specification provides another graph model task deployment apparatus supporting a multi-engine framework, comprising at least one storage medium and at least one processor, the at least one storage medium storing computer instructions; the at least one processor is configured to execute the computer instructions to implement another graph model task implementation method supporting a multi-engine framework as described above.
Drawings
The present description will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not intended to be limiting, and in these embodiments like numerals are used to indicate like structures, wherein:
FIG. 1 is a schematic diagram of an application scenario of a diagram learning system supporting a multi-engine framework according to some embodiments of the present description;
FIG. 2 is a block diagram of a diagram learning system supporting a multi-engine framework in accordance with some embodiments of the present description;
FIG. 3 is a block diagram of a diagram learning system supporting a multi-engine framework according to further embodiments of the present description;
FIG. 4 is an exemplary flow diagram of graph learning in support of a multi-engine framework, shown in accordance with some embodiments of the present description;
FIG. 5 is a schematic diagram of a terminal interface for a task configuration shown in some embodiments herein;
FIG. 6 is an exemplary flow diagram illustrating graph learning in support of a multi-engine framework according to further embodiments of the present description.
Detailed Description
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only examples or embodiments of the present description, and that for a person skilled in the art, the present description can also be applied to other similar scenarios on the basis of these drawings without inventive effort. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.
It should be understood that "system", "device", "unit" and/or "module" as used in this specification is a method for distinguishing different components, elements, parts or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.
As used in this specification and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.
Flow charts are used in this description to illustrate operations performed by a system according to embodiments of the present description. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to or removed from these processes.
FIG. 1 is a schematic diagram of an application scenario of a graph model task implementation system supporting a multi-engine framework, according to some embodiments of the present description.
The scenario 100 may relate to various scenarios in which a machine learning model (which may be referred to as a Graph model or a Graph neural network model, such as a Graph Convolution Network (GCN), a Graph Attention network (Graph Attention network), a Graph autoencoder (Graph Autoencoders), a Graph generation network (Graph generation Networks), and the like) suitable for Graph learning is applied, such as scenarios in which a Graph model required for sample data training is used, and prediction tasks such as entity relationship prediction and entity classification are performed through a trained Graph model based on input data.
A machine learning engine (e.g., a deep learning engine such as a tensrflow or a Pytorch) is an engine that can provide a computation structure component (e.g., a structure component such as a convolutional layer, a fully-connected layer, a transform, and other layers of structures, a function for building a layer structure, and a computation function component such as model modeling, model training, and model prediction) of a machine learning model and implement a machine learning model task (e.g., a model training task or a model prediction task). The machine learning engine may be deployed on a processing device.
The graph learning framework can be a learning framework built on a machine learning engine and is specially used for providing calculation structural components suitable for the graph model and realizing the task of the graph model. For example, the graph learning framework may build a computing structure component suitable for a graph learning model based on a more general or underlying computing structure component provided by a machine learning engine, and provide a computing function component suitable for model training or prediction of the graph model, model prediction, and the like.
In some embodiments, the graph model task may include tasks such as training of the graph model, prediction (e.g., prediction tasks such as inter-entity relationship prediction in a knowledge graph, entity classification, etc.).
In some embodiments, a user may provide graph data to a processing device, and implement graph model tasks such as graph model training, graph model prediction and the like by using a machine learning engine on the processing device or a graph learning framework built based on the machine learning engine.
In some embodiments, a machine learning engine or a graph learning framework on the processing device includes graph models supported by the machine learning engine or the graph learning framework. In some embodiments, a graph model in a machine learning engine or graph learning framework on a processing device may be built by calling a computing structure component on the machine learning engine or graph learning framework based on user instructions. In some embodiments, the graph models included in the machine learning engine or graph learning framework on the processing device may be obtained from elsewhere (e.g., other platforms) and imported into the machine learning engine or graph learning framework on the processing device. The user may "flood" the graph data to a machine learning engine or a graph learning framework on the processing device, and may then process the graph data through the graph model on the machine learning engine or the graph learning framework of the processing device to implement the graph model task.
The machine learning engine may include a variety of types, such as Tensorflow, Pyorch, and the like. Each engine may support the establishment of one or more image Learning frameworks, such as, for example, Pytorch-based image Learning frameworks Pytorch Geometric (PyG) and Deep Graph Library (DGL), and tenswaf-based image Learning frameworks Ant Graph Learning (AGL) and tenswaf GNN (TF-GNN), among others. Different machine learning engines and image learning frameworks can have respective characteristics and advantages. The input data formats, model codes, model training modes, task deployment modes and the like of different machine learning engines and graph learning frames can be different. Due to the foregoing differences, when some graph model tasks need to be implemented by using any one or more of multiple machine learning engines and multiple graph learning frames, or some graph model tasks need to be migrated between multiple machine learning engines or multiple graph learning frames, input data, models, task configurations, task deployments, and the like need to be re-adapted for different machine learning engines or different graph learning frames, for example, input data, model codes, model training modes, task deployment modes, and the like need to be rewritten, which brings great complexity to business applications. Illustratively, a user obtains a graph model built and trained based on Pythrch on another platform (e.g., corresponding to a first processing device), and if the graph model needs to be transferred to another platform (e.g., corresponding to a second processing device) for model prediction, and the other platform only supports a Tensorflow engine, the graph model can not be directly used on the other platform. At this point, the user may need to rebuild, train the graph model based on the Tensorflow engine. For another example, when a user obtains certain image data and needs to perform correlation prediction using a graph convolution network in a DGL framework on a certain platform, the user is likely to be required to rewrite the format of the image data to obtain an image data format that can be supported by the DGL framework.
In view of this, some embodiments of the present disclosure provide a graph model task implementation method or system supporting multiple engine frames, which integrates multiple machine learning engines and multiple graph learning frames supported by different engines, and implements a graph model task by obtaining graph data in a unified preset data format, converting the graph data from the preset data format into a data format supported by a target machine learning engine (specified by a user) or a target graph learning frame, and providing target input data obtained by converting the data format to the target machine learning engine or the target graph learning frame. From the perspective of a user, in the embodiments of the present description, after receiving graph data in a preset data format, a processing device may select a target machine learning engine or a target graph learning framework from two or more integrated machine learning engines and/or two or more integrated graph learning frameworks to implement a graph model task, implement a graph model task supporting multiple engine frameworks, and reduce the labor cost and the technical cost used by the user.
As shown in fig. 1, the scenario 100 may include a user end 110, a network 120, and a processing device 130.
In some embodiments, the user terminal 110 may include various processing devices such as a computer, a personal computer, or a computing platform formed by connecting a plurality of computers in various structures. In some embodiments, the user terminal 110 may be implemented on a cloud platform. For example, the cloud platform may include one or a combination of private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, cross-cloud, multi-cloud, and the like. In some embodiments, the user end 110 can be one or more users, can be a user directly using the graph model task implementation system supporting the multi-engine framework, and can also be other related users. In some embodiments, the user terminal 110 may communicate information and data with other components (e.g., the processing device 130) in the scenario 100.
In some embodiments, the network 120 can facilitate information and/or data exchange. In some embodiments, one or more components of the scenario 100 (e.g., the user terminal 110, the processing device 130) may exchange information and/or data with one or more components of the scenario 100 via the network 120. The network 120 may include one or more of a public network (e.g., the internet), a private network (e.g., a Local Area Network (LAN), a Wide Area Network (WAN)), etc.), a wired network (e.g., ethernet), a wireless network (e.g., an 802.11 network, a wireless Wi-Fi network, etc.), a cellular network (e.g., a Long Term Evolution (LTE) network), a frame relay network, a Virtual Private Network (VPN), a satellite network, a telephone network, a router, a hub, a server computer, etc. In some embodiments, the network 120 may be any one or more of a wired network or a wireless network.
In some embodiments, the processing device 130 may be a system having computing and processing capabilities. The processing device 130 may include various computers, such as a server, a personal computer, or may be a computing platform (or computing cluster) formed by a plurality of computers connected in various configurations. In some embodiments, the processing device 130 may be implemented on a cloud platform. For example, the cloud platform may include one or a combination of private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, cross-cloud, multi-cloud, and the like. The processing device 130 may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core processing device). By way of example only, Processing device 130 may include various common general purpose Central Processing Units (CPUs), Graphics Processing Units (GPUs), microprocessors, application-specific integrated circuits (ASICs), or other types of integrated circuits. One or more machine learning engines and/or one or more graph learning frameworks may be deployed on the processing device 130, and implement the graph model task implementation method and system supporting multiple engine frameworks according to some embodiments of the present specification.
FIG. 2 is a block diagram of a graph model task processing system supporting a multi-engine framework, according to some embodiments of the present description.
In some embodiments, the graph model task processing system 200 supporting a multi-engine framework may be implemented on a processing device (e.g., processing device 130).
In some embodiments, the system 200 may include a first graph data acquisition module 210, an engine or framework determination module 220, a data adaptation module 230, and a graph model task implementation processing module 240. In some embodiments, the system 200 may further include a first configuration information receiving module 250.
In some embodiments, the first graph data obtaining module 210 may be configured to obtain graph data in a preset data format. In some embodiments, the predetermined data format supports feature recording of one or more of the following graph data: directed graph, undirected graph, multiple graph, heteromorphic graph, timing diagram. In some embodiments, the predetermined data format supports sparse storage of matrices and/or vectors. In some embodiments, the graph data in the preset data format is stored in a python numpy array storage format.
In some embodiments, the engine or framework determination module 220 may be used to determine a target machine learning engine or target graph learning framework from more than two machine learning engines and/or more than two graph learning frameworks. In some embodiments, the machine learning engine comprises one or more of: tensorflow, Pyorch. In some embodiments, the image learning framework comprises any plurality of: pyroch Geometric (PyG), Deep Graph Library (DGL), Ant Graph Learning (AGL), Tensorflow GNN (TF-GNN).
In some embodiments, the data adaptation module 230 may be configured to convert the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning framework, resulting in target input data.
In some embodiments, graph model task implementation processing module 240 may be configured to provide the target input data to the target machine learning engine or the target graph learning framework, through which the target machine learning engine or the target graph learning framework processes the target input data based on its supported machine learning models to implement the graph model task. In some embodiments, a machine learning engine is used to provide the computational structural components of the machine learning model and to enable training and/or prediction of the machine learning model. In some embodiments, the graph learning framework builds on a machine learning engine and is used to provide computational structure components of a machine learning model suitable for graph learning and to enable training and/or prediction of a machine learning model suitable for graph learning. In some embodiments, the graph model task includes training or prediction of a machine learning model suitable for graph learning. In some embodiments, the target machine learning engine or the machine learning model supported by the target graph learning framework is obtained by importing or by calling a computing structure component provided by the target machine learning engine or the target graph learning framework.
In some embodiments, the first configuration information receiving module 250 may be used to receive task configuration information. In some embodiments, the task configuration information includes one or more of the following information: training iteration times, a verification interval of a model training stage, sample repeated training times, machine learning model storage addresses after training is completed, and machine learning model loading addresses.
In some embodiments, the graph model task implementation processing module 240 may be further configured to invoke or configure the target machine learning engine or the target graph learning framework to complete the graph learning model task based on the task configuration information.
FIG. 3 is a block diagram of a graph model task deployment system supporting a multi-engine framework in accordance with further embodiments of the present description.
In some embodiments, the graph model task implementation system 300 supporting a multi-engine framework may be implemented on a processing device (e.g., processing device 130). In some embodiments, one processing device may deploy both system 200 and system 300 simultaneously. In some embodiments, system 200 and system 300 may also be deployed by 2 processing devices, respectively, according to business needs.
In some embodiments, the system 300 may include a second graph data acquisition module 310, a second configuration information receiving module 320, and a task submission module 330.
In some embodiments, the second graph data obtaining module 310 may be configured to obtain the graph data in a preset data format and a machine learning model supported by the target machine learning engine or the target graph learning framework.
In some embodiments, the second configuration information receiving module 320 may be configured to receive task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning framework from more than two machine learning engines and/or more than two graph learning frameworks and appointing a task deployment mode.
In some embodiments, the task submission module 330 may be configured to submit the graph data in a preset data format and the machine learning model to one or more computing nodes based on the task deployment manner, so that the computing nodes implement the graph model task based on the method (e.g., the method 400) described in some embodiments of the present disclosure. In some embodiments, the computing nodes may comprise local computing nodes or cloud computing nodes.
It should be understood that the illustrated system and its modules may be implemented in a variety of ways. For example, in some embodiments, the system and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein the hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory for execution by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the methods and systems described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided, for example, on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules in this specification may be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also by software executed by various types of processors, for example, or by a combination of the above hardware circuits and software (e.g., firmware).
It should be noted that the above description of the system and its modules is for convenience of description only and should not limit the present disclosure to the illustrated embodiments. It will be appreciated by those skilled in the art that, given the teachings of the present system, any combination of modules or sub-system configurations may be used to connect to other modules without departing from such teachings.
FIG. 4 is an exemplary flow diagram of a graph model task processing method supporting a multiple engine framework according to some embodiments of the present description.
In some embodiments, method 400 may be performed by a processing device (e.g., processing device 130). In some embodiments, method 400 may be implemented by a graph model task processing system 200 supporting a multi-engine framework deployed on a processing device (e.g., processing device 130).
As shown in fig. 4, the method 400 may include:
in step 410, graph data in a preset data format is obtained.
In some embodiments, this step 410 may be performed by the first data acquisition module 210.
Graph data is data having a data structure of Graph (Graph), and data is represented and stored by nodes and edges connecting the nodes in the Graph data. The knowledge graph is a typical application of graph data, and the description will mainly describe the graph data, and the explanation of the related terms of the knowledge graph is also applicable to other graph data without specific description.
A knowledge graph refers to a knowledge base consisting of a series of nodes representing entities and edges representing relationships between the entities. There may be multiple types of nodes, called node types, for indicating various types of entities. Edges in the graph represent relationships. Edges may also be of various types, called edge types, for indicating various types of relationships. The entity is an extensive abstraction of an objective individual, and may refer to a tangible object in the physical world, such as a natural person, a merchant, and the like, or an intangible object, such as a payment account, a wifi account, and the like, or a more abstract concept, such as nationality, gender, and the like. Relationships may be used to express a connection between different entities, such as a person "living in" Beijing, Zhang three "administers" merchant A, account A "transfers" to account B, and so on.
The nodes/edges may have attributes. It will be appreciated that the attributes of a node are attributes of the entity represented by the node, and the edge attributes are attributes of the relationship represented by the edge. For example, the attribute of a node representing a certain drug may include a taking method, a taking frequency, a course of treatment, contraindication information, etc. of the drug, and the attribute representing the relationship of "having been in service" may include a length of service. Wherein the attribute comprises an attribute value.
The knowledge graph may include directed graphs, undirected graphs, multiple graphs, heterogeneous graphs, timing diagrams, and the like. Wherein: directed graphs refer to knowledge-graphs in which edges may be directional, and directional edges may be unidirectional or bidirectional to indicate directionality of relationships. Undirected graphs refer to the fact that edges in a knowledge graph can be undirected. When the knowledge-graph is an undirected graph, edges may indicate that a relationship has no directionality or that a relationship is bidirectional (e.g., "friends," "follow each other," etc. relationships). The multiple graph means that edges between the same two nodes in the knowledge graph can comprise a plurality of edges, and the plurality of edges between the nodes can represent various relation types between the nodes. An anomaly graph refers to a knowledge graph having multiple types of nodes and/or edges. The time sequence diagram refers to that nodes and edges in the knowledge graph have time sequence, and information is transmitted among the nodes according to the time sequence of the nodes and/or the edges.
In a knowledge graph, an edge that points to a node may be referred to as an in-edge of the node, and an edge that points from (i.e., points to) a node may be referred to as an out-edge of the node. A bidirectional or undirected edge may be referred to as an ingress edge of any node to which the edge connects. In some embodiments, an incoming edge and an outgoing edge may also refer to a node's outgoing edge and incoming edge. For a directed graph, an edge attribute may include the direction of the edge.
In the knowledge graph, the number of edges connected by one node can be expressed in degrees, and the number of edges connected by one node to other nodes can also be expressed. The degree may include out degree, which refers to the number of edges from the node pointing to other nodes, and in degree, which refers to the number of edges pointing to the node. In some embodiments, there may be hotspots in the knowledge-graph, which are nodes whose value of the indexing (e.g., out-degree or in-degree) is greater than a threshold.
In some embodiments, the graph data may be stored or recorded in different data formats. The format of the graph data can be the storage content and form of the appointed node and the storage content and form of the appointed edge. The preset data format may be used to define a particular graph data format so that the graph data acquired by the processing device has a uniform format. In some embodiments, the specific format of the preset data is not particularly limited as long as the diagram data can be stored and recorded.
In some embodiments, the user end 110 may directly obtain the graph data in the preset data format from another platform or device, or the user end 110 converts the graph data in another format or the original example data into the graph data in the preset data format according to a standard or a specification of the preset data format, or the user end 110 may generate the graph data in the preset data format based on the graph data in another format or the original example data through a platform providing a data format conversion service.
In still other embodiments, a new default data format may be designed such that it is compatible with storing or recording feature records of one or more of the following graph data: directed graph, undirected graph, multiple graph, heteromorphic graph, timing diagram.
In some embodiments, the preset data format may represent one or more of the following attribute information of the nodes in the graph data in a matrix and/or a vector: node id (e.g., central node id, number of sub-graph node in original graph data, etc.), node feature, node timestamp, node type (in some embodiments, multiple types of information of a node may be represented by a matrix), mapping of node type name to type id (where, type id may represent a position number, such as a column number, of a corresponding node type in a node type representation matrix/vector), corresponding position number of a node feature name in a feature representation matrix/vector of a corresponding node (e.g., a "gender" attribute of a node includes two values of "male" and "female", and "male" and "female" may be respectively id-encoded by a one hot method to obtain an attribute value of 0 or 1, and respectively stored in 10 th column and 21 th column of the node feature matrix), total number of nodes in the sub-graph, and edges of node connection (which may include an edge, a node connection, etc.) Edge entry), degree of a node (which may include out degree and in degree), graph data slice id where the node is located, and the like.
In some embodiments, the preset data format may represent one or more of the following attribute information of an edge in the graph data in a matrix and/or a vector: the edge feature, the edge timestamp, the edge type (in some embodiments, multiple types of information of one edge may be represented by a matrix), a mapping of an edge type name to a type id (where the type id may represent a position number, such as a column number, of a corresponding edge type in an edge type representation matrix/vector), a position number corresponding to the edge feature name in a feature representation matrix/vector of a corresponding edge (for example, a "payment relation" attribute of an edge includes two values of "pay" and "collect", and the "pay" and "collect" may be respectively id-encoded by a one hot method to obtain an attribute value of 0 or 1, and respectively stored in the 10 th column and the 21 st column of the edge feature matrix), a total number of edges in the sub-graph, and the like.
Through the embodiment, different graph data such as a directed graph, an undirected graph, a multiple graph, an abnormal graph, a timing graph and the like can be recorded in the preset data format. For example, the preset data format may support undirected graph feature information records such as node features, edge features, node types, edge types, and the like; the preset data format can support the record of directed graph characteristic information such as the outgoing edge and the incoming edge of a node; the preset data format can support the recording of multiple graph characteristic information such as multiple edges between two nodes by supporting the edges connected with the recording nodes; the preset data format can support the node to have a plurality of types of information and the node to have a heterogeneous graph characteristic information record of the plurality of types of information; the predetermined data format may support the time stamp profile characteristic information records such as node time stamps, side time stamps, etc.
In some embodiments, the data of a node or edge may be sparse, e.g., more elements 0 than non-0 elements. The preset data format can support sparse storage of the matrix and/or the vector, wherein the sparse storage refers to only storing elements with values different from 0 in the matrix and/or the vector and positions of the elements. By the embodiment, the memory occupation of the input data can be reduced, and the memory resource is saved.
In some embodiments, the graph data in the preset data format may be stored at the bottom layer in a python numpy array storage format. Because various machine learning engines and various image learning frames can support the storage format of the python numpy array, the embodiment can realize that the image data file is commonly used on various machine learning engines and various image learning frames, thereby avoiding the situation that the content of the image data file needs to be copied for storing again because the machine learning engines and/or the image learning frames do not support the storage format of the file, further simplifying the implementation method of the image model task supporting the multi-engine frame and saving the computing resources.
In some embodiments, this step 420 may be performed by the engine or framework determination module 220.
In some embodiments, a machine learning engine integrated on a processing device or in a graph model task implementation system supporting a multi-engine framework as set forth in this specification may include one or more of: tensorflow, Pyorch. A machine learning engine selected from two or more machine learning engines may be referred to as a target machine learning engine.
In some embodiments, two or more graph learning frameworks integrated on a processing device or within a graph model task implementation system supporting a multi-engine framework as set forth in this specification can include any of the following: pyroch Geometric (PyG), Deep Graph Library (DGL), Ant Graph Learning (AGL), Tensorflow GNN (TF-GNN). A picture learning frame selected from two or more picture learning frames may be referred to as a target picture learning frame.
In some embodiments, the target machine learning engine and/or target learning framework may be determined automatically by the processing device (e.g., the processing device determines according to some preset rules). In some embodiments, the target machine learning engine and/or the target learning framework may be determined by a user, for example, the user may determine the target machine learning engine and/or the target learning framework via a user terminal, and the user terminal may transmit information of the target machine learning engine and/or the target learning framework to the processing device. Further, the user may select the target machine learning engine or the target learning frame from a user interface provided by the processing device to the user side, or the user may directly send description information of the target machine learning engine or the target learning frame to the processing device.
In some embodiments, this step 430 may be performed by the data adaptation module 230.
The graph data formats supported by different machine learning engines or graph learning frameworks may be different. In some embodiments, the graph data may be converted from a preset data format to a data format corresponding to a target machine learning engine or target graph learning framework. For example, different data adapters are configured to respectively implement conversion of graph data in a preset data format into different other formats. The corresponding data adapter may be determined based on a preset data format and a data format that needs to be converted (for example, a data format corresponding to the pytorech engine is converted from the preset data format). The data adapter may be a software module implemented by an operator or a functional component, and may be written according to a standard or specification of two formats before and after conversion. In some embodiments, the graph data in the preset data format may be input into a data adapter, and the data adapter performs data conversion on the graph data in the preset data format to obtain the graph data in the required data format (i.e., the data format corresponding to the target machine learning engine or the target graph learning framework). The graph data of a desired data format obtained by the data conversion may be referred to as target input data.
In some embodiments, this step 440 may be performed by the graph model task realization processing module 240.
In some embodiments, after obtaining the target input data, the target input data may be provided to a target machine learning engine or a target graph learning framework on the processing device, and the target machine learning engine or the target graph learning framework may process the target input data through a machine learning model supported by the target machine learning engine or the target graph learning framework to implement the graph model task. Specifically, the target machine learning engine or the target graph learning framework may train the machine learning model based on the target input data, or use the target input data as the input data of the machine learning model to obtain the relevant prediction result. For more details about the obtaining method of the machine learning model on the target machine learning engine or the target graph learning framework and the graph model task, reference may be made to fig. 1 and its related description, which are not repeated herein.
In some embodiments, method 400 may further include the step of receiving task configuration information for configuring the graph model task. The task configuration information may specify relevant control information in a model training or prediction process. In some embodiments, the task configuration information may be determined by a user and provided to the processing device. In some embodiments, the model training process or the dominant completion model prediction process may be controlled by the graph model task realization processing module 240 in the system 200 based on user configured task configuration information. The graph model task implementation processing module 240 can be regarded as an independent functional module independent of the engines or the frameworks, and can dominate or coordinate the graph model task processing flow under each engine or framework.
Fig. 5 is a schematic diagram of a task configuration terminal interface in which a user can set task configuration information and provide the task configuration information to a processing device through a user terminal according to some embodiments of the present disclosure. As shown in the interface diagram in fig. 5, in particular, the task configuration information may include one or more of the following information: training iteration times (for example, the training iteration times are set to 8000, or-1, when the value is-1, the training iteration times of the model are not limited, or the model is trained based on other preset iteration times instead of the times set in the task configuration information), a verification interval of a model training stage (for example, the verification interval is set to 2000, the model is verified once after every 2000 training rounds), a sample repeated training time (for example, the verification interval is set to 1, and each sample is used for 1 time during training), and other model training relevant configuration information, and a machine learning model storage address (not shown in the figure) after training, a machine learning model loading address (not shown in the figure, when a model needs to be used, for example, when a model prediction task is implemented, a trained model can be called/obtained from the machine learning model loading address), and the like.
In some embodiments, the machine learning model load address and the trained machine learning model store address may be the same. For example, a machine learning model is built on a machine learning engine or a graph learning framework, the machine learning model is trained and then stored in a machine learning model storage address after training, and the address can be used as a machine learning model loading address to call/acquire the trained model from the address when the machine learning model is used. In some embodiments, the machine learning model may be obtained from elsewhere (e.g., other platform) and imported into the processing device to implement the graph model task based on the machine learning model, where the machine learning model is loaded with a memory address of the machine learning model elsewhere, which may be different from the memory address of the machine learning model after training on the current device is completed. For example, the graph model task implementation processing module 240 may obtain a graph model under a trained DGL framework from a machine learning model loading address, load the graph model into the DGL framework, and input target input data under a DGL format into the graph model to obtain a corresponding prediction result.
In some embodiments, the user may also select a target machine learning engine or target graph learning framework from the terminal interface shown in fig. 5. Taking FIG. 5 as an example, what the user currently selects is the pytorch engine.
In some embodiments, the user may also specify the task deployment in the terminal interface shown in fig. 5. For example, the number of workers (computing nodes) in the interface of fig. 5 is configured to be 1, which indicates that the task deployment mode is standalone deployment, and when the number is greater than 1, it may indicate that the task deployment mode is distributed deployment. In some embodiments, the user may also select a local computing node or a remote (cloud) computing node to perform graph model tasks via the terminal interface shown in fig. 5.
In some embodiments, the information input by the user through fig. 5 may be collectively referred to as task configuration information, and the user terminal 110 may transmit the task configuration information to the processing device 130 through the network 120.
The method 400 implements a process of a graph model task, and in some embodiments, the graph model task implementation method or system supporting a multi-engine framework provided in some embodiments of the present disclosure may further implement submission or distribution of the graph model task, so as to deploy the graph model task to one or more computing nodes.
FIG. 6 is an exemplary flow diagram of a graph model task deployment method supporting a multi-engine framework according to some embodiments of the present description.
In some embodiments, a processing device may be capable of performing the functionality of method 400 or the functionality of method 600. In some embodiments, multiple processing devices may be provided, with one processing device serving as a central node for performing method 600 and the remaining processing devices serving as multiple computing nodes for implementing method 400, depending on business needs. Of course, in some embodiments, some of the processing devices are only capable of implementing the method 400 and another portion of the processing devices are only capable of implementing the method 600.
As shown in fig. 6, the method 600 may include:
In some embodiments, this step 610 may be performed by the second graph data acquisition module 310.
The method for obtaining the graph data in the preset data format may refer to step 410 and the related description thereof, which are not described herein again.
The method for obtaining the machine learning model supported by the target machine learning engine or the target graph learning framework may refer to fig. 1 and 4 and their related descriptions.
In some embodiments, this step 620 may be performed by the second configuration information receiving module 320.
In some embodiments, the task configuration information received here may include the selection results of the user through fig. 5 regarding the number of computing nodes and the local or cloud nodes. In some embodiments, the number of computing nodes and the selection result of the local or cloud node may also be configured through another independent terminal interface. In some embodiments, the user may also configure, through the terminal interface shown in fig. 5, the number of CPUs of each worker (computing node), the number of memories of each worker, and other computing resource configuration information of the processing device (not shown in the figure).
In some embodiments, this step 630 may be performed by the task submission module 330.
As described above, the task configuration information such as the number of computing nodes may indicate a task deployment manner, and the task submission module 330 may submit the graph data in the preset data format and the machine learning model to one or more computing nodes (the computing nodes may include processing devices) according to the task deployment manner.
For example, graph data in a preset data format and the machine learning model may be submitted to one computing node (for example, when worker is 1) or to a plurality of computing nodes (for example, when worker is an integer greater than 1) by the task submitting module 330 according to a task deployment manner. The task submission module 330 on the processing device as a compute node may interface with the task submission module 330 on the central node to receive task information submitted by the central node. It is understood that the task submission module 330 may serve as a task submission layer to implement task deployment interaction between computing nodes.
In some embodiments, the task submitting module 330 may also submit the model-related configuration information input by the user through the terminal interface and the model training-related configuration information to one or more computing nodes at the same time, so that the computing nodes may implement the graph model task through the method shown in fig. 4 based on the configuration information.
In some embodiments, the computing nodes may include a local node and a cloud computing node. And furthermore, the graph model task can realize single machine or cluster deployment and can also realize local or cloud deployment.
In some embodiments, when graph data in a preset data format and a machine learning model are submitted to a plurality of computing nodes according to a task deployment manner, a central node may split the graph data into a plurality of fragments, and submit the plurality of fragments to the plurality of computing nodes, respectively, and the graph model data may be repeatedly distributed to the plurality of computing nodes, so that the plurality of computing nodes may complete a graph model prediction task in parallel. In some embodiments, the central node may also distribute a plurality of graph data to a plurality of compute nodes to enable joint training of the graph model. In some embodiments, the central node may also split and distribute the graph model. Regarding the distribution manner of graph data and/or graph models among a plurality of computing nodes, which may be determined according to a specific business scenario, some embodiments of the present description do not perform any limitation.
It should be noted that the above descriptions of the processes and methods are only for illustration and description and do not limit the scope of the application of the present specification. Various modifications and alterations to the procedures and methods will be apparent to those skilled in the art in light of this description. However, such modifications and variations are intended to be within the scope of the present description. For example, the order of steps in the processes and methods may be altered, steps in different processes and methods may be combined, and the like.
The embodiment of the present specification further provides a graph model task processing apparatus supporting a multi-engine framework, including at least one storage medium and at least one processor, where the at least one storage medium is used for storing computer instructions; the at least one processor is configured to execute the computer instructions to implement a graph model task implementation method supporting a multi-engine framework, which may include: acquiring graph data in a preset data format; determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames; converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning framework to obtain target input data; and providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on the machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
The embodiment of the present specification further provides another graph model task deployment apparatus supporting a multi-engine framework, including at least one storage medium and at least one processor, where the at least one storage medium is used for storing computer instructions; the at least one processor is configured to execute the computer instructions to implement another graph model task implementation supporting a multi-engine framework, and the method may include: acquiring graph data in a preset data format and a machine learning model supported by a target machine learning engine or a target graph learning framework; receiving task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames and appointing a task deployment mode; and submitting graph data in a preset data format and the machine learning model to one or more computing nodes based on a task deployment mode so that the computing nodes realize graph model tasks based on the graph model task implementation method supporting the multi-engine framework.
The beneficial effects that may be brought by the embodiments of the present description include, but are not limited to: (1) according to the method provided by the specification, graph data in a unified preset data format is obtained, then the graph data are converted from the preset data format into a data format corresponding to a target machine learning engine or a target graph learning frame determined from more than two machine learning engines and/or more than two graph learning frames, and target input data obtained after the data format is converted are provided for the target machine learning engine or the target graph learning frame to achieve a graph model task. After the graph data in the preset data format is provided for the processing equipment, the processing equipment can realize the graph model task based on more than two machine learning engines and/or one or more target machine learning engines or target graph learning frames in more than two graph learning frames, and the complexity of realizing the graph model task supporting a multi-engine frame is reduced; (2) by the method provided by the specification, the unified task configuration information is determined for the graph model task, the graph model task can be realized by various machine learning models and/or various graph learning frameworks based on the unified task configuration information, the complexity of supporting the graph model task of a multi-engine framework is further reduced, and meanwhile, the flexibility of task deployment is increased. It is to be noted that different embodiments may produce different advantages, and in different embodiments, any one or combination of the above advantages may be produced, or any other advantages may be obtained.
Having thus described the basic concept, it will be apparent to those skilled in the art that the foregoing detailed disclosure is to be regarded as illustrative only and not as limiting the present specification. Various modifications, improvements and adaptations to the present description may occur to those skilled in the art, although not explicitly described herein. Such modifications, improvements and adaptations are proposed in the present specification and thus fall within the spirit and scope of the exemplary embodiments of the present specification.
Also, the description uses specific words to describe embodiments of the description. Reference throughout this specification to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the specification is included. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, some features, structures, or characteristics of one or more embodiments of the specification may be combined as appropriate.
Moreover, those skilled in the art will appreciate that aspects of the present description may be illustrated and described in terms of several patentable species or situations, including any new and useful combination of processes, machines, manufacture, or materials, or any new and useful improvement thereof. Accordingly, aspects of this description may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the present description may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media.
The computer storage medium may comprise a propagated data signal with the computer program code embodied therewith, for example, on baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination. A computer storage medium may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer storage medium may be propagated over any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or any combination of the preceding.
Computer program code required for the operation of various portions of this specification may be written in any one or more of a variety of programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C + +, C #, VB.NET, Python, and the like, a conventional programming language such as C, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP, a dynamic programming language such as Python, Ruby, and Groovy, or other programming languages, and the like. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or processing device. In the latter scenario, the remote computer may be connected to the user's computer through any network format, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as a software as a service (SaaS).
Additionally, the order in which the elements and sequences of the process are recited in the specification, the use of alphanumeric characters, or other designations, is not intended to limit the order in which the processes and methods of the specification occur, unless otherwise specified in the claims. While various presently contemplated embodiments of the invention have been discussed in the foregoing disclosure by way of example, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments herein. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by software-only solutions, such as installing the described system on an existing processing device or mobile device.
Similarly, it should be noted that in the preceding description of embodiments of the present specification, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not intended to imply that more features than are expressly recited in a claim. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.
Numerals describing the number of components, attributes, etc. are used in some embodiments, it being understood that such numerals used in the description of the embodiments are modified in some instances by the use of the modifier "about", "approximately" or "substantially". Unless otherwise indicated, "about", "approximately" or "substantially" indicates that the number allows a variation of ± 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that may vary depending upon the desired properties of the individual embodiments. In some embodiments, the numerical parameter should take into account the specified significant digits and employ a general digit preserving approach. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the range are approximations, in the specific examples, such numerical values are set forth as precisely as possible within the scope of the application.
For each patent, patent application publication, and other material, such as articles, books, specifications, publications, documents, etc., cited in this specification, the entire contents of each are hereby incorporated by reference into this specification. Except where the application history document does not conform to or conflict with the contents of the present specification, it is to be understood that the application history document, as used herein in the present specification or appended claims, is intended to define the broadest scope of the present specification (whether presently or later in the specification) rather than the broadest scope of the present specification. It is to be understood that the descriptions, definitions and/or uses of terms in the accompanying materials of this specification shall control if they are inconsistent or contrary to the descriptions and/or uses of terms in this specification.
Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present disclosure. Other variations are also possible within the scope of the present description. Thus, by way of example, and not limitation, alternative configurations of the embodiments of the specification can be considered consistent with the teachings of the specification. Accordingly, the embodiments of the present description are not limited to only those embodiments explicitly described and depicted herein.
Claims (14)
1. A graph model task processing method supporting a multi-engine framework, the method comprising:
acquiring graph data in a preset data format;
determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames;
converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning frame to obtain target input data;
and providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on the machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
2. The method of claim 1, the preset data format supporting feature recording of one or more of the following graph data: directed graph, undirected graph, multiple graph, heteromorphic graph, timing diagram.
3. The method according to claim 1, the preset data format supporting sparse storage of matrices and/or vectors.
4. The method of claim 1, further comprising storing the graph data in the preset data format in a python numpy array storage format.
5. The method of claim 1, the machine learning engine to provide a computational structural component of the machine learning model and to enable training and/or prediction of the machine learning model; the graph learning framework is built on a machine learning engine and is used for providing a calculation structure component of a machine learning model suitable for graph learning and realizing the training and/or prediction of the machine learning model suitable for graph learning;
the graph model task includes training or prediction of a machine learning model suitable for graph learning.
6. The method of claim 1, wherein the machine learning model supported by the target machine learning engine or the target graph learning framework is obtained by importing or by calling a computing structure component provided by the target machine learning engine or the target graph learning framework.
7. The method of claim 1, further comprising:
receiving task configuration information; the task configuration information includes one or more of the following information: training iteration times, a verification interval of a model training stage, sample repeated training times, a machine learning model storage address after training is completed, and a machine learning model loading address;
and calling or configuring the target machine learning engine or the target graph learning framework to complete the graph model task based on the task configuration information.
8. The method of claim 1, the machine learning engine comprising one or more of: tensorflow, Pyorch; the image learning framework includes any number of: pyroch Geometric (PyG), Deep Graph Library (DGL), Ant Graph Learning (AGL), Tensorflow GNN (TF-GNN).
9. A graph model task processing system supporting a multi-engine framework, comprising:
the first image data acquisition module is used for acquiring image data in a preset data format;
the engine or frame determining module is used for determining a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames;
the data adaptation module is used for converting the graph data from the preset data format into a data format corresponding to the target machine learning engine or the target graph learning frame to obtain target input data;
and the graph model task realization processing module is used for providing the target input data to the target machine learning engine or the target graph learning framework, and processing the target input data through the target machine learning engine or the target graph learning framework based on the machine learning model supported by the target machine learning engine or the target graph learning framework to realize a graph model task.
10. A graph model task processing apparatus supporting a multi-engine framework, comprising at least one storage medium and at least one processor, the at least one storage medium for storing computer instructions; the at least one processor is configured to execute the computer instructions to implement the method of any of claims 1-8.
11. A graph model task deployment method supporting a multi-engine framework comprises the following steps:
acquiring graph data in a preset data format and a machine learning model supported by a target machine learning engine or a target graph learning framework;
receiving task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames and appointing a task deployment mode;
submitting graph data in a preset data format and the machine learning model to one or more computing nodes based on a task deployment mode so that the computing nodes can realize graph model tasks based on the method of any one of claims 1 to 7.
12. The method of claim 11, the computing node comprising a local computing node or a cloud computing node.
13. A graph model task deployment system supporting a multi-engine framework, comprising:
the second graph data acquisition module is used for acquiring graph data in a preset data format and a machine learning model supported by a target machine learning engine or a target graph learning framework;
the second configuration information receiving module is used for receiving task configuration information; the task configuration information is used for appointing a target machine learning engine or a target graph learning frame from more than two machine learning engines and/or more than two graph learning frames and appointing a task deployment mode;
the task submitting module is used for submitting graph data in a preset data format and the machine learning model to one or more computing nodes based on the task deployment mode so that the computing nodes can realize graph model tasks based on the method of any one of claims 1 to 7.
14. A graph model task deployment apparatus supporting a multi-engine framework, comprising at least one storage medium and at least one processor, the at least one storage medium for storing computer instructions; the at least one processor is configured to execute the computer instructions to perform the method of any of claims 11-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210393596.0A CN114511100B (en) | 2022-04-15 | 2022-04-15 | Graph model task implementation method and system supporting multi-engine framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210393596.0A CN114511100B (en) | 2022-04-15 | 2022-04-15 | Graph model task implementation method and system supporting multi-engine framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114511100A true CN114511100A (en) | 2022-05-17 |
| CN114511100B CN114511100B (en) | 2023-01-13 |
Family
ID=81554858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210393596.0A Active CN114511100B (en) | 2022-04-15 | 2022-04-15 | Graph model task implementation method and system supporting multi-engine framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114511100B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116185532A (en) * | 2023-04-18 | 2023-05-30 | 之江实验室 | Task execution system, method, storage medium and electronic equipment |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150050623A1 (en) * | 2011-09-01 | 2015-02-19 | L-3 Communications Corporation | Adaptive training system, method and apparatus |
| WO2015134044A1 (en) * | 2014-03-07 | 2015-09-11 | L-3 Communications Corporation | Adaptive training system, method and apparatus |
| CN107423817A (en) * | 2017-04-17 | 2017-12-01 | 星环信息科技（上海）有限公司 | The method and apparatus that a kind of deep learning is realized |
| CN109816114A (en) * | 2018-12-29 | 2019-05-28 | 大唐软件技术股份有限公司 | A kind of generation method of machine learning model, device |
| CN110119271A (en) * | 2018-12-19 | 2019-08-13 | 厦门渊亭信息科技有限公司 | A kind of model across machine learning platform defines agreement and adaption system |
| CN110737805A (en) * | 2019-10-18 | 2020-01-31 | 网易（杭州）网络有限公司 | Method and device for processing graph model data and terminal equipment |
| CN111275199A (en) * | 2020-01-17 | 2020-06-12 | 深圳壹账通智能科技有限公司 | Conversion method and system of deep learning model file, computer equipment and computer readable storage medium |
| CN111400000A (en) * | 2020-03-09 | 2020-07-10 | 百度在线网络技术（北京）有限公司 | Network request processing method, device, equipment and storage medium |
| CN113379070A (en) * | 2021-08-13 | 2021-09-10 | 苏州浪潮智能科技有限公司 | Deep learning framework conversion method, system, storage medium and equipment |
| CN113505895A (en) * | 2021-08-05 | 2021-10-15 | 上海高德威智能交通系统有限公司 | Machine learning engine service system, model training method and configuration method |
| CN114185519A (en) * | 2021-12-08 | 2022-03-15 | 中盈优创资讯科技有限公司 | Method and device for optimizing Spark and machine learning framework integration |
| CN114358136A (en) * | 2021-12-10 | 2022-04-15 | 鹏城实验室 | Image data processing method and device, intelligent terminal and storage medium |
-
2022
- 2022-04-15 CN CN202210393596.0A patent/CN114511100B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150050623A1 (en) * | 2011-09-01 | 2015-02-19 | L-3 Communications Corporation | Adaptive training system, method and apparatus |
| WO2015134044A1 (en) * | 2014-03-07 | 2015-09-11 | L-3 Communications Corporation | Adaptive training system, method and apparatus |
| CN107423817A (en) * | 2017-04-17 | 2017-12-01 | 星环信息科技（上海）有限公司 | The method and apparatus that a kind of deep learning is realized |
| CN110119271A (en) * | 2018-12-19 | 2019-08-13 | 厦门渊亭信息科技有限公司 | A kind of model across machine learning platform defines agreement and adaption system |
| CN109816114A (en) * | 2018-12-29 | 2019-05-28 | 大唐软件技术股份有限公司 | A kind of generation method of machine learning model, device |
| CN110737805A (en) * | 2019-10-18 | 2020-01-31 | 网易（杭州）网络有限公司 | Method and device for processing graph model data and terminal equipment |
| CN111275199A (en) * | 2020-01-17 | 2020-06-12 | 深圳壹账通智能科技有限公司 | Conversion method and system of deep learning model file, computer equipment and computer readable storage medium |
| CN111400000A (en) * | 2020-03-09 | 2020-07-10 | 百度在线网络技术（北京）有限公司 | Network request processing method, device, equipment and storage medium |
| CN113505895A (en) * | 2021-08-05 | 2021-10-15 | 上海高德威智能交通系统有限公司 | Machine learning engine service system, model training method and configuration method |
| CN113379070A (en) * | 2021-08-13 | 2021-09-10 | 苏州浪潮智能科技有限公司 | Deep learning framework conversion method, system, storage medium and equipment |
| CN114185519A (en) * | 2021-12-08 | 2022-03-15 | 中盈优创资讯科技有限公司 | Method and device for optimizing Spark and machine learning framework integration |
| CN114358136A (en) * | 2021-12-10 | 2022-04-15 | 鹏城实验室 | Image data processing method and device, intelligent terminal and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| KATERINA DOKA等: "IReS: Intelligent, Multi-Engine Resource Scheduler for Big Data Analytics Workflows", 《2015 ACM SIGMOD INTERNATIONAL CONFERENCE ON MANAGEMENT OF DATA》 * |
| KATERINA DOKA等: "Mix ‘n’ match multi-engine analytics", 《2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA)》 * |
| 夏冉: "基于Spark的机器学习Web服务引擎设计", 《指挥控制与仿真》 * |
| 赵惠芳: "多分析引擎集成应用关键技术研究", 《中国安全防范技术与应用》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116185532A (en) * | 2023-04-18 | 2023-05-30 | 之江实验室 | Task execution system, method, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114511100B (en) | 2023-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210150330A1 (en) | Systems and methods for machine learning based modeling | |
| CN112799850A (en) | Model training method, model prediction method, and model control system | |
| US11514263B2 (en) | Method and apparatus for processing image | |
| CN111815169B (en) | Service approval parameter configuration method and device | |
| CN111177400A (en) | Associated display method and device of equipment, service and data based on knowledge graph | |
| CN110879776A (en) | Test case generation method and device | |
| CN114511100B (en) | Graph model task implementation method and system supporting multi-engine framework | |
| CN111311480A (en) | Image fusion method and device | |
| CN115794913A (en) | Data processing method and device in artificial intelligence system | |
| CN110288625B (en) | Method and apparatus for processing image | |
| CN113592033B (en) | Oil tank image recognition model training method, oil tank image recognition method and device | |
| CN111111179A (en) | Unity-based graphics resource loading method and device | |
| CN111985471A (en) | License plate positioning method and device and storage medium | |
| CN115022328B (en) | Server cluster, testing method and device of server cluster and electronic equipment | |
| US11775264B2 (en) | Efficient deployment of machine learning and deep learning model's pipeline for serving service level agreement | |
| CN113762421B (en) | Classification model training method, flow analysis method, device and equipment | |
| CN112416488B (en) | User portrait implementing method, device, computer equipment and computer readable storage medium | |
| WO2021068529A1 (en) | Image recognition method and apparatus, computer device and storage medium | |
| CN114185871A (en) | Data migration method, data migration device, medium, and electronic device | |
| CN110598056A (en) | Node layout determination method and device | |
| CN113098961A (en) | Component uploading method, device and system, computer equipment and readable storage medium | |
| CN111353585A (en) | Structure searching method and device of neural network model | |
| CN114707680B (en) | Aircraft 3D model generation method and device, electronic equipment and readable medium | |
| CN116630436B (en) | Camera external parameter correction method, camera external parameter correction device, electronic equipment and computer readable medium | |
| CN115544622B (en) | Urban and rural participated three-dimensional planning design platform, method, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |