CN112784996B - Machine learning method and system based on graph representation - Google Patents
Machine learning method and system based on graph representation Download PDFInfo
- Publication number
- CN112784996B CN112784996B CN202011637601.5A CN202011637601A CN112784996B CN 112784996 B CN112784996 B CN 112784996B CN 202011637601 A CN202011637601 A CN 202011637601A CN 112784996 B CN112784996 B CN 112784996B
- Authority
- CN
- China
- Prior art keywords
- machine learning
- framework
- socket
- data
- graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Testing And Monitoring For Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a machine learning method and system based on graph representation. Wherein the method comprises the following steps: the first device pre-processes the original data through the machine learning framework; the first device controls the machine learning framework to transmit the preprocessed data to the graph characterization framework through the first socket and the second socket; the second device performs graph characterization processing on the processed data through the graph characterization framework; the second device controls the diagram sign framework to transmit the diagram sign obtained through diagram sign processing to the machine learning framework through the second socket and the first socket; the first device controls the machine learning framework to process the object to which the original data belongs based on the chart characteristics. The performance of the algorithm model in machine learning can be improved.
Description
Technical Field
The present invention relates to the field of machine learning technologies, and in particular, to a machine learning method and system based on graph representation.
Background
In machine learning, an algorithm model obtained through machine learning may be processed according to the representation of the object to be processed based on the learned mapping relationship between the representation and the processing result, so as to obtain the processing result of the object to be processed, where the processing may refer to any one of classification processing, clustering processing, detection processing, regression processing, and the like. Thus, the accuracy of the characterization will directly impact the performance of the algorithm model.
Since there are often some associations between objects, for example, one person may be a colleague with another person, one phone number may call another phone number, a vehicle may be affiliated with one person, etc. Therefore, the characteristics of the object to be reflected in the related art can be represented by the graph characteristics, which can be calculated through graph characteristic processing such as graph embedding algorithm, graph neural network algorithm and the like.
However, conventional machine learning frameworks often do not have the capability of processing the symptoms, so that they cannot use the symptoms as the object's representation, resulting in poor performance of the algorithm model in machine learning.
Disclosure of Invention
The embodiment of the invention aims to provide a machine learning method and a system based on graph representation, so as to improve the performance of an algorithm model in machine learning. The specific technical scheme is as follows:
in a first aspect of the embodiment of the present invention, a machine learning method based on graph representation is provided, and the machine learning method is applied to a machine learning system, where the machine learning system includes a first device and a second device, the first device is equipped with a machine learning framework, the second device is equipped with a graph representation framework, the machine learning framework is provided with a first socket, the graph representation framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the method comprises the following steps:
the first device pre-processes the original data through the machine learning framework;
the first device controls the machine learning framework to transmit the preprocessed data to the graph characterization framework through the first socket and the second socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework;
the second device controls the diagram sign framework to transmit the diagram sign obtained through diagram sign processing to the machine learning framework through the second socket and the first socket;
the first device controls the machine learning framework to process the object to which the original data belongs based on the chart characteristics.
In one possible embodiment, the second device controls the first device to transmit the first device to the machine learning framework via the first socket and the second socket, including:
the second device controls the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real-time via the second socket and the first socket.
In a possible embodiment, the first device controls the machine learning framework to process the object to which the original data belongs based on the chart feature, including:
the first device controls the machine learning framework to process the object to which the original data belongs in real time based on the received graph representation.
In one possible embodiment, the second device includes a GPU, and the graphics framework runs on the GPU of the second device.
In one possible embodiment, the first device controls the machine learning framework to transmit the preprocessed processed data to the graph characterization framework via the first socket and the second socket, including:
the first device controls the machine learning framework to perform format conversion on the processed data obtained through preprocessing to obtain first data to be transmitted in a preset third format, wherein the third format is a data format supported by the first socket and the second socket;
the first device controls the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework, including:
the second device controls the diagram representation framework to perform format conversion on the first data to be transmitted to obtain diagram representation input data with a preset second format, wherein the second format is a data format supported by the diagram representation framework;
and the second equipment performs graph characterization processing on the graph characterization input data through the graph characterization framework.
In one possible embodiment, the second device controls the first device to transmit the first device to the machine learning framework via the first socket and the second socket, including:
the second device controls the diagram characterization framework to perform format conversion on the diagram characterization obtained through diagram characterization processing to obtain second data to be transmitted in a preset third format;
the second device controls the chart characterization framework to send the second data to be transmitted to the first socket through the second socket;
the first device controls the machine learning framework to process the object to which the original data belongs based on the chart sign, and the method comprises the following steps:
the first device controls the machine learning framework to perform format conversion on the second data to be transmitted to obtain machine learning input data with a preset first format, wherein the first format is a data format supported by the machine learning framework;
the first device controls the machine learning framework to process the object to which the original data belongs based on the machine learning input data.
In a second aspect of the embodiment of the present invention, there is provided a machine learning system based on graph representation, the machine learning system including a first device carrying a machine learning framework and a second device carrying a graph representation framework, the machine learning framework being provided with a first socket, the graph representation framework being provided with a second socket, the first socket and the second socket supporting the same data format and interfacing;
the first device is used for preprocessing the original data through the machine learning framework; controlling the machine learning framework to transmit the preprocessed data to the chart framework through the first socket and the second socket;
the second device is used for performing graph characterization processing on the processed data through the graph characterization framework; controlling the diagram sign framework to transmit diagram signs obtained through diagram sign processing to the machine learning framework through the second socket and the first socket;
the first device is further used for controlling the machine learning framework to process the object to which the original data belongs based on the chart characteristics.
In a possible embodiment, the second device is specifically configured to control the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real time via the second socket and the first socket.
In a possible embodiment, the first device is specifically configured to control the machine learning framework to process, in real time, the object to which the original data belongs based on the received graph representation.
In one possible embodiment, the second device includes a GPU, and the graphics framework runs on the GPU of the second device.
In a possible embodiment, the first device is specifically configured to control the machine learning framework to perform format conversion on the preprocessed data to obtain first data to be transmitted in a preset third format, where the third format is a data format supported by the first socket and the second socket; controlling the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device is specifically configured to control the diagram characterization framework to perform format conversion on the first data to be transmitted, so as to obtain diagram characterization input data with a preset second format, where the second format is a data format supported by the diagram characterization framework; and performing graph characterization processing on the graph characterization input data through the graph characterization framework.
In a possible embodiment, the second device is specifically configured to control the diagram feature framework to perform format conversion on the diagram feature obtained through the diagram feature processing, so as to obtain second data to be transmitted in a preset third format; controlling the chart characterization framework to send the second data to be transmitted to the first socket through the second socket;
the first device is specifically configured to control the machine learning framework to perform format conversion on the second data to be transmitted, so as to obtain machine learning input data with a preset first format, where the first format is a data format supported by the machine learning framework; and controlling the machine learning framework to process the object to which the original data belongs based on the machine learning input data.
The embodiment of the invention has the beneficial effects that:
according to the machine learning method and system based on graph representation, the inter-program communication between the technical machine learning framework and the graph representation framework can be realized through the socket, so that data between the two frameworks are completed, graph representation output by the graph representation framework can be input into the machine learning framework, and therefore the machine learning framework without graph representation processing capability can also perform machine learning based on the graph representation, and the performance of an algorithm model in the machine learning is improved.
Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a machine learning method based on graph representation according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data flow according to an embodiment of the present invention;
FIG. 3a is a timing diagram of a machine learning method based on graph representation according to an embodiment of the present invention;
FIG. 3b is a schematic timing diagram of another machine learning method based on graph representation according to an embodiment of the present invention;
FIG. 3c is a schematic diagram of another timing diagram of a machine learning method based on graph characterization according to an embodiment of the present invention;
FIG. 3d is a schematic timing diagram of another machine learning method based on graph representation according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a machine learning system according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
For more clear explanation of the machine learning method based on graph representation provided by the embodiment of the present invention, an exemplary explanation will be given below of one possible application scenario of the machine learning method based on graph representation provided by the embodiment of the present invention, it will be understood that the following example is only one possible application scenario of the machine learning method based on graph representation provided by the embodiment of the present invention, and in other possible embodiments, the machine learning method based on graph representation provided by the embodiment of the present invention may be applied to other possible application scenarios.
It is assumed that for practical purposes, a classification model for determining whether a person is a legitimate person needs to be trained. It will be appreciated that although each of the legitimate and illegitimate persons has some characteristics, it is theoretically possible to distinguish between the legitimate and illegitimate persons using the characteristics for representing the characteristics of each person themselves, the persons do not exist independently, for example, there may be association relationships between each person and other objects than the person, such as person-to-person, person-to-vehicle, person-to-phone number, etc., and these association relationships may also reflect to some extent whether the person is a legitimate person, for example, if there is an association relationship between a person and a plurality of legitimate persons, the likelihood that the person is a legitimate person is high, and also for example, if there is an association relationship between a person and an illegitimate IP address, the likelihood that the person is an illegitimate person is high.
Therefore, the characterization can effectively reflect the association relationship between the person and other objects except the person, so that the model obtained through training can synthesize the characteristics of multiple dimensions to judge whether the person is a legal person or not, and the accuracy of the judgment result is improved.
In order to enable the representation to effectively reflect the association relationship between the person and other objects except the person, a relationship graph between each person and other objects except the person can be constructed, and the graph representation of the relationship graph is obtained by performing graph representation processing such as graph embedding algorithm, graph neural network algorithm and the like on the relationship graph, wherein the graph representation can be regarded as a vectorized representation of the relationship graph. The chart feature can effectively reflect the association relationship between the person and other objects except the person.
However, the conventional machine learning framework does not have the capability of processing graph characterization, so in one possible solution provided by the embodiment of the present invention, in order to enable the machine learning framework to perform machine learning based on graph characterization, it may be to separately construct a machine learning framework, such as a Spark (an open source cluster computing environment) framework, and a graph characterization framework, such as a GNN (Graph Neural Networks, graph neural network) framework, which has the capability of graph characterization, and separately place the machine learning framework and the graph characterization framework in two different devices, where the device provided with the machine learning framework is referred to as a first device and the device provided with the graph characterization framework is referred to as a second device for convenience of description hereinafter.
The data for representing the characteristics of each person and the association relationship between each person and other objects may be input to the first device, where the first device performs preprocessing, such as extraction, cleaning, conversion, and other preprocessing, on the data by using a machine learning framework to obtain preprocessed data, where the first device stores the preprocessed data in a storage system local to the first device, where the machine learning framework is a Spark, for example, and may be stored in a distributed file system local to the first device.
The first device sends the locally stored preprocessed data to the second device via a network with the second device, and the second device stores the received preprocessed data in a storage system local to the second device.
The graph characterization framework in the second device reads the stored preprocessed data from the storage system local to the second device, so as to perform graph characterization processing on the preprocessed data, obtain graph characterizations capable of being used for representing the self characteristics of each person and the association relationship between each person and other objects, and store the obtained graph characterizations in the storage system local to the second device.
The second device sends the locally stored profile to the first device over a network with the first device, and the first device stores the received profile in a storage system local to the first device. The machine learning framework in the first device reads the stored chart sign from a storage system local to the first device, performs machine learning based on the read chart sign, and trains to obtain a classification model for judging whether the person is legal or not.
However, in this solution, it takes a certain time to read and write data between the machine learning framework and the storage system local to the first device, read and write data between the graph characterization framework and the storage system local to the second device, and perform data interaction between the first device and the second device through the network, which results in lower efficiency of machine learning.
Based on this, the embodiment of the present invention provides a machine learning method based on graph representation, and referring to fig. 1, fig. 1 is a schematic flow diagram of the machine learning method based on graph representation provided by the embodiment of the present invention, which may include:
s101, the first device preprocesses original data through a machine learning framework.
S102, the first device controls the machine learning framework to transmit the preprocessed data to the chart characterization framework through the first socket and the second socket.
S103, the second device performs graph characterization processing on the processed data through the graph characterization framework.
S104, the second device control diagram representation framework transmits the diagram representation obtained through the diagram representation processing to the machine learning framework through the second socket and the first socket.
S105, the first device controls the machine learning framework to process the object to which the original data belongs based on the graph representation.
By adopting the embodiment, the inter-program communication between the technical machine learning framework and the schematic feature framework can be realized through the socket, so that the data between the two frameworks are completed, the schematic feature output by the schematic feature framework can be input into the machine learning framework, the machine learning framework without schematic feature processing capability can also perform machine learning based on the schematic feature, and the performance of an algorithm model in the machine learning is improved.
In S101, the first device is a device on which the machine learning framework is mounted, and may be one device or may be formed by integrating a plurality of devices, and may be a physical device or a virtual device, which is not limited in this example.
The preprocessing may include one or more of extraction processing, cleaning processing, and conversion processing, and may include other processing according to the application scenario, which is not limited in this embodiment.
The original data may be data for representing the own characteristics of each person and the association relationship between each person and other objects, and may be data for representing the own characteristics of each vehicle and the association relationship between each vehicle and other objects, for example, assuming that a model for classifying vehicles needs to be trained. The objects herein may include one or more of personnel, vehicles, telephone numbers, MAC addresses, IMSI numbers, locations, face images, license plates, etc.
In S102, the graph characterization framework is disposed in the second device, the first socket is disposed in the machine learning framework, and the second socket is disposed in the graph characterization framework. The first socket and the second socket support the same data format and interface, i.e., the machine learning framework and the graph characterization framework may enable inter-program communication through the first socket and the second socket.
The first socket and the second socket support the same data format, which means that the same data format is used for data transmission in a transmission protocol of the first socket and the second socket, hereinafter, for convenience of description, the data format supported by the machine learning framework is referred to as a first data format, the data format supported by the chart characterization framework is referred to as a second data format, and the data format supported by the first socket and the second socket is referred to as a third data format.
In one possible implementation, to increase the transmission rate, the processed data may be compressed, and the compressed processed data is transmitted to the graph characterization framework via the first socket and the second socket.
In S103, the graph characterization process may be a process using a graph embedding algorithm or a graph neural network algorithm, and the type of graph characterization process should be a graph characterization process type supported by the graph characterization framework. For example, if the graph characterization framework is a knowledge graph embedding framework, the graph characterization process is to process with a graph embedding algorithm, and if the graph characterization framework is a graph neural network framework, the graph characterization process is to process with a graph neural network algorithm. Since the graph characterization process is not a major improvement point of the present invention, the detailed flow of the graph characterization process is not described herein.
In one possible embodiment, to increase the efficiency of the graph characterization process by the graph characterization framework, the second device may include a GPU, where the graph characterization framework runs on the GPU. By adopting the embodiment, the efficiency of graph characterization processing of the graph characterization framework can be effectively improved by utilizing the strong graph operation capability of the GPU.
In S104, in one possible implementation, to increase the transmission rate, the graph symbols may be compressed, and the compressed graph symbols are transmitted to the machine learning framework via the second socket and the first socket.
In S105, the object to which the original data belongs may refer to an acquisition object when the original data is acquired, and by way of example, if the original data is acquired by acquiring each person, the object to which the original data belongs is a person.
If the original data is the sample data obtained by collecting the sample object, the processing result obtained by processing the object to which the original data belongs can be used for model training, and if the original data is the data obtained by collecting the object to be processed, the processing result obtained by processing the object to which the original data belongs is the processing result of the object to be processed.
The machine learning method based on graph representation provided by the embodiment of the invention can be applied to a scene for training an algorithm model, and can also be used in a scene for processing an object by using the algorithm model after training to obtain the algorithm model.
The processing herein may refer to classification processing, detection processing, clustering processing, regression processing, and other processing than classification, detection, clustering, and regression, and is not limited in this embodiment.
For a clearer description of the graph-representation-based machine learning method provided by the embodiment of the present invention, reference may be made to fig. 2, where fig. 2 is a schematic diagram illustrating a data flow provided by the embodiment of the present invention, and a thin solid line with an arrow indicates a data flow in the graph-representation-based machine learning method provided by the embodiment of the present invention, and a thick solid line with an arrow indicates a data flow in the machine learning method described in the foregoing application scenario.
Therefore, in the machine learning method based on graph representation provided by the embodiment of the invention, the direct communication between the machine learning frame and the graph representation frame is realized through the socket, so that the reading and writing between the machine learning frame and the storage system of the first device and the reading and writing between the graph representation frame and the storage system of the second device are omitted, the machine learning efficiency can be effectively improved, and the landing of the algorithm model in the machine learning can be accelerated.
In a possible embodiment, the foregoing steps S101-S105 may be performed serially, i.e. S101 is performed after completion of S102, S102 is performed after completion of S103, and so on, where the timing diagram of these five steps may be as shown in fig. 3 a.
Assuming that the duration consumed by S101 is t1, the duration consumed by S102 is t2, the duration consumed by S103 is t3, the duration consumed by S104 is t4, and the duration consumed by S105 is t5, executing the machine learning method based on graph representation according to the timing chart shown in fig. 3a consumes a total of t1+t2+t3+t4+t5.
In some possible embodiments, the total time consumed to execute the machine learning method based on graph representation provided by the embodiment of the invention can be reduced by parallel execution. It can be understood that, in the graph characterization process, in order to enable the obtained graph characterization to represent the association relationship between the object and other objects as comprehensively as possible, the graph characterization process needs to be performed based on the complete processed data, that is, S103 needs to be performed after the execution of S102 is completed.
The symptoms are represented in the form of vectors in which the values of the elements are determined at different moments, i.e. from the first element of the vector to the last element of the vector will last for a period of time, and from the first element of the vector, execution of S104 can already be started. Thus in one possible embodiment, the second device may be a control graph representation framework transmitting the generated graph representation to the machine learning framework via the second socket and the first socket in real time.
At this time, the timing chart of the machine learning method based on graph representation provided by the embodiment of the present invention may be shown in fig. 3 b. As shown in fig. 3b, it can be seen that the total duration consumed by executing the graph-representation-based machine learning method provided by the embodiment of the present invention according to the timing chart shown in fig. 3b is significantly less than t1+t2+t3+t4+t5, i.e., the machine learning efficiency can be effectively improved by selecting the embodiment.
It will be appreciated that not all of the features characterized based on the graph characterization may be required to be processed, and thus in one possible embodiment, the machine learning framework may already begin executing S105 when it receives the graph characterization, i.e., the first device may control the machine learning framework to process the object to which the raw data belongs in real-time based on the received graph characterization.
At this time, the timing chart of the machine learning method based on graph representation provided by the embodiment of the invention may be shown in fig. 3 c. As shown in fig. 3c, it can be seen that the total duration consumed by executing the graph-representation-based machine learning method provided by the embodiment of the present invention according to the timing chart shown in fig. 3c is significantly less than t1+t2+t3+t4+t5, i.e., the machine learning efficiency can be effectively improved by selecting the embodiment.
In yet another possible embodiment, the second device may be a control graph representation framework transmitting the generated graph representation to the machine learning framework via the second socket and the first socket in real time, and the first device may control the machine learning framework to process the object to which the original data belongs in real time based on the received graph representation.
At this time, the timing chart of the machine learning method based on graph representation provided by the embodiment of the invention may be shown in fig. 3 d. As shown in fig. 3d, it can be seen that the total duration consumed by executing the graph-representation-based machine learning method provided by the embodiment of the invention according to the timing chart shown in fig. 3d is significantly less than t1+t2+t3+t4+t5, i.e., the machine learning efficiency can be effectively improved by selecting the embodiment.
It will be appreciated that the first data format supported by the machine learning framework and the second data format supported by the diagramming framework tend to be different.
Therefore, when the machine learning framework transmits the processed data to the graph characterization framework, the first device may control the machine learning framework to perform format conversion on the processed data obtained through preprocessing, so as to obtain first data to be transmitted in a third format. And the first device controls the machine learning framework to send the first data to be transmitted to the second socket through the first socket.
After the diagram representation framework receives the first data to be transmitted through the second socket, the second equipment controls the diagram representation framework to perform format conversion on the first data to be transmitted to obtain diagram representation input data in a second format, and diagram representation processing is performed on the diagram representation input data through the diagram representation framework. It is understood that the diagrammatical input data is obtained by converting the format of the first data to be transmitted, and the first data to be transmitted is obtained by converting the format of the processed data, so that the diagrammatical input data is treated as diagrammatical data.
Similarly, when the schematic representation frame transmits the schematic representation data to the machine learning frame, the second device may control the machine learning frame to perform format conversion on the schematic representation obtained through the schematic representation processing, so as to obtain second data to be transmitted in a third format. And the second device control diagram representation framework sends the second data to be transmitted to the first socket through the second socket.
After the machine learning framework receives the second data to be transmitted through the first socket, the first equipment can control the machine learning framework to perform format conversion on the second data to be transmitted to obtain machine learning input data in a first format, and the machine learning framework processes objects to which the original data belong based on the machine learning input data. It can be understood that the machine learning input data is obtained by format conversion of the second data to be transmitted, and the second data to be transmitted is obtained by format conversion of the graph representation, so that the processing of the object to which the original data belongs based on the machine learning input data can be regarded as the processing of the object to which the original data belongs based on the graph representation.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a machine learning system based on graph representation provided by an embodiment of the present invention, where the machine learning system includes a first device 410 and a second device 420, the first device is equipped with a machine learning framework 411, the second device is equipped with a graph representation framework 421, the machine learning framework is provided with a first socket, the graph representation framework is provided with a second socket, and the first socket and the second socket support the same data format and are in docking;
the first device 410 is configured to pre-process raw data through the machine learning frame 411; control the machine learning framework 411 to transmit the preprocessed data to the graph characterization framework 421 via the first socket and the second socket;
the second device 420 is configured to perform graph characterization processing on the processed data through the graph characterization framework 421; control the diagram feature framework 421 to transmit the diagram feature obtained through the diagram feature processing to the machine learning framework 411 via the second socket and the first socket;
the first device 410 is further configured to control the machine learning framework 411 to process an object to which the original data belongs based on the chart feature.
In a possible embodiment, the second device 420 is specifically configured to control the graph characterization framework 421 to transmit the generated graph characterization to the machine learning framework 411 in real time via the second socket and the first socket.
In a possible embodiment, the first device 410 is specifically configured to control the machine learning framework 411 to process, in real time, the object to which the original data belongs based on the received graph representation.
In one possible embodiment, the second device 420 includes a GPU, and the graphics framework 421 runs on the GPU of the second device.
In a possible embodiment, the first device 410 is specifically configured to control the machine learning framework 411 to perform format conversion on the processed data obtained through the preprocessing, so as to obtain first data to be transmitted in a preset third format, where the third format is a data format supported by the first socket and the second socket; controlling the machine learning framework 411 to transmit the first data to be transmitted to the second socket through the first socket;
the second device 420 is specifically configured to control the diagram representation framework 421 to perform format conversion on the first data to be transmitted, so as to obtain diagram input data with a preset second format, where the second format is a data format supported by the diagram representation framework 421; the graph characterization input data is processed by the graph characterization framework 421.
In a possible embodiment, the second device 420 is specifically configured to control the graph characterization framework 421 to perform format conversion on the graph characterization obtained through the graph characterization process, so as to obtain second data to be transmitted in a preset third format; controlling the graph characterization framework 421 to send the second data to be transmitted to the first socket through the second socket;
the first device 410 is specifically configured to control the machine learning framework 411 to perform format conversion on the second data to be transmitted, so as to obtain machine learning input data with a preset first format, where the first format is a data format supported by the machine learning framework 411; the machine learning framework 411 is controlled to process the object to which the original data belongs based on the machine learning input data.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (12)
1. A machine learning method based on graph characterization is characterized by being applied to a machine learning system, wherein the machine learning system comprises a first device and a second device, the first device is provided with a machine learning framework, the second device is provided with a graph characterization framework, the machine learning framework is provided with a first socket, the graph characterization framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the method comprises the following steps:
the first device pre-processes the original data through the machine learning framework;
the first device controls the machine learning framework to transmit the preprocessed data to the graph characterization framework through the first socket and the second socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework;
the second device controls the diagram sign framework to transmit the diagram sign obtained through diagram sign processing to the machine learning framework through the second socket and the first socket;
the first device controls the machine learning framework to process the object to which the original data belongs based on the chart characteristics.
2. The method of claim 1, wherein the second device controlling the machine learning framework to transmit the graph symptoms processed by the graph symptoms to the machine learning framework via the second socket and the first socket comprises:
the second device controls the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real-time via the second socket and the first socket.
3. The method of claim 2, wherein the first device controlling the machine learning framework to process the object to which the raw data belongs based on the diagrammatical features comprises:
the first device controls the machine learning framework to process the object to which the original data belongs in real time based on the received graph representation.
4. The method of claim 1, wherein the second device comprises a GPU, and wherein the graphics framework runs on the GPU of the second device.
5. The method of claim 1, wherein the first device controlling the machine learning framework to transmit the preprocessed processed data to the graph characterization framework via the first socket and the second socket comprises:
the first device controls the machine learning framework to perform format conversion on the processed data obtained through preprocessing to obtain first data to be transmitted in a preset third format, wherein the third format is a data format supported by the first socket and the second socket;
the first device controls the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework, including:
the second device controls the diagram representation framework to perform format conversion on the first data to be transmitted to obtain diagram representation input data with a preset second format, wherein the second format is a data format supported by the diagram representation framework;
and the second equipment performs graph characterization processing on the graph characterization input data through the graph characterization framework.
6. The method of claim 1, wherein the second device controlling the machine learning framework to transmit the graph symptoms processed by the graph symptoms to the machine learning framework via the second socket and the first socket comprises:
the second device controls the diagram characterization framework to perform format conversion on the diagram characterization obtained through diagram characterization processing to obtain second data to be transmitted in a preset third format;
the second device controls the chart characterization framework to send the second data to be transmitted to the first socket through the second socket;
the first device controls the machine learning framework to process the object to which the original data belongs based on the chart sign, and the method comprises the following steps:
the first device controls the machine learning framework to perform format conversion on the second data to be transmitted to obtain machine learning input data with a preset first format, wherein the first format is a data format supported by the machine learning framework;
the first device controls the machine learning framework to process the object to which the original data belongs based on the machine learning input data.
7. A machine learning system based on graph characterization, characterized in that the machine learning system comprises a first device and a second device, wherein the first device is provided with a machine learning framework, the second device is provided with a graph characterization framework, the machine learning framework is provided with a first socket, the graph characterization framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the first device is used for preprocessing the original data through the machine learning framework; controlling the machine learning framework to transmit the preprocessed data to the chart framework through the first socket and the second socket;
the second device is used for performing graph characterization processing on the processed data through the graph characterization framework; controlling the diagram sign framework to transmit diagram signs obtained through diagram sign processing to the machine learning framework through the second socket and the first socket;
the first device is further used for controlling the machine learning framework to process the object to which the original data belongs based on the chart characteristics.
8. The system of claim 7, wherein the second device is specifically configured to control the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real time via the second socket and the first socket.
9. The system of claim 8, wherein the first device is specifically configured to control the machine learning framework to process the object to which the raw data belongs in real time based on the received graph representation.
10. The system of claim 7, wherein the second device comprises a GPU, and wherein the graphics framework is operative on the GPU of the second device.
11. The system of claim 7, wherein the first device is specifically configured to control the machine learning framework to perform format conversion on the preprocessed data to obtain first data to be transmitted in a preset third format, where the third format is a data format supported by the first socket and the second socket; controlling the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device is specifically configured to control the diagram characterization framework to perform format conversion on the first data to be transmitted, so as to obtain diagram characterization input data with a preset second format, where the second format is a data format supported by the diagram characterization framework; and performing graph characterization processing on the graph characterization input data through the graph characterization framework.
12. The system of claim 7, wherein the second device is specifically configured to control the graph characterization framework to perform format conversion on the graph characterization obtained through the graph characterization processing, so as to obtain second data to be transmitted in a preset third format; controlling the chart characterization framework to send the second data to be transmitted to the first socket through the second socket;
the first device is specifically configured to control the machine learning framework to perform format conversion on the second data to be transmitted, so as to obtain machine learning input data with a preset first format, where the first format is a data format supported by the machine learning framework; and controlling the machine learning framework to process the object to which the original data belongs based on the machine learning input data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637601.5A CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637601.5A CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112784996A CN112784996A (en) | 2021-05-11 |
| CN112784996B true CN112784996B (en) | 2023-06-02 |
Family
ID=75754984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011637601.5A Active CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112784996B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2870571A2 (en) * | 2012-07-09 | 2015-05-13 | Toyota Motor Europe NV/SA | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| CN111245673A (en) * | 2019-12-30 | 2020-06-05 | 浙江工商大学 | SDN time delay sensing method based on graph neural network |
| CN111899150A (en) * | 2020-08-28 | 2020-11-06 | Oppo广东移动通信有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111914180A (en) * | 2020-08-19 | 2020-11-10 | 腾讯科技（深圳）有限公司 | User characteristic determination method, device, equipment and medium based on graph structure |
| CN111930518A (en) * | 2020-09-22 | 2020-11-13 | 北京东方通科技股份有限公司 | Knowledge graph representation learning-oriented distributed framework construction method |
| CN112035261A (en) * | 2020-09-11 | 2020-12-04 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
| CN112069412A (en) * | 2020-09-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information recommendation method and device, computer equipment and storage medium |
| CN112069398A (en) * | 2020-08-24 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information pushing method and device based on graph network |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2495265A (en) * | 2011-07-07 | 2013-04-10 | Toyota Motor Europe Nv Sa | Artificial memory system for predicting behaviours in order to assist in the control of a system, e.g. stability control in a vehicle |
| US11132604B2 (en) * | 2017-09-01 | 2021-09-28 | Facebook, Inc. | Nested machine learning architecture |
| US11144812B2 (en) * | 2017-09-01 | 2021-10-12 | Facebook, Inc. | Mixed machine learning architecture |
| US20190073580A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Sparse Neural Network Modeling Infrastructure |
| US20190287032A1 (en) * | 2018-03-16 | 2019-09-19 | International Business Machines Corporation | Contextual Intelligence for Unified Data Governance |
-
2020
- 2020-12-31 CN CN202011637601.5A patent/CN112784996B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2870571A2 (en) * | 2012-07-09 | 2015-05-13 | Toyota Motor Europe NV/SA | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| CN111245673A (en) * | 2019-12-30 | 2020-06-05 | 浙江工商大学 | SDN time delay sensing method based on graph neural network |
| CN111914180A (en) * | 2020-08-19 | 2020-11-10 | 腾讯科技（深圳）有限公司 | User characteristic determination method, device, equipment and medium based on graph structure |
| CN112069398A (en) * | 2020-08-24 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information pushing method and device based on graph network |
| CN111899150A (en) * | 2020-08-28 | 2020-11-06 | Oppo广东移动通信有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112035261A (en) * | 2020-09-11 | 2020-12-04 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
| CN112069412A (en) * | 2020-09-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information recommendation method and device, computer equipment and storage medium |
| CN111930518A (en) * | 2020-09-22 | 2020-11-13 | 北京东方通科技股份有限公司 | Knowledge graph representation learning-oriented distributed framework construction method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112784996A (en) | 2021-05-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109639481B (en) | Deep learning-based network traffic classification method and system and electronic equipment | |
| CN107578017B (en) | Method and apparatus for generating image | |
| CN110958218B (en) | Data transmission method based on multi-network communication and related equipment | |
| CN109214238A (en) | Multi-object tracking method, device, equipment and storage medium | |
| CN109413023B (en) | Training of machine recognition model, machine recognition method and device, and electronic equipment | |
| US20140336869A1 (en) | Automating Predictive Maintenance for Automobiles | |
| CN111985504B (en) | Copying detection method, device, equipment and medium based on artificial intelligence | |
| CN112149615A (en) | Face living body detection method, device, medium and electronic equipment | |
| CN113037783B (en) | Abnormal behavior detection method and system | |
| CN109635993A (en) | Operation behavior monitoring method and device based on prediction model | |
| CN113033966A (en) | Risk target identification method and device, electronic equipment and storage medium | |
| CN110119621B (en) | Attack defense method, system and defense device for abnormal system call | |
| CN112784996B (en) | Machine learning method and system based on graph representation | |
| CN114282692A (en) | Model training method and system for longitudinal federal learning | |
| Fathurrahman et al. | Lightweight convolution neural network for image-based malware classification on embedded systems | |
| CN113361455B (en) | Training method of face counterfeit identification model, related device and computer program product | |
| CN112669353B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN115049843A (en) | Confrontation sample generation method and device, electronic equipment and storage medium | |
| CN114840421A (en) | Log data processing method and device | |
| CN114898191A (en) | Hand-held fabric fiber component nondestructive cleaning analyzer and method | |
| CN113673476A (en) | Face recognition model training method and device, storage medium and electronic equipment | |
| CN113821794A (en) | Distributed trusted computing system and method | |
| CN112420146A (en) | Information security management method and system | |
| KR20210038027A (en) | Method for Training to Compress Neural Network and Method for Using Compressed Neural Network | |
| CN105991450A (en) | MAC address table updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |