CN112784996A - Machine learning method and system based on graph representation - Google Patents
Machine learning method and system based on graph representation Download PDFInfo
- Publication number
- CN112784996A CN112784996A CN202011637601.5A CN202011637601A CN112784996A CN 112784996 A CN112784996 A CN 112784996A CN 202011637601 A CN202011637601 A CN 202011637601A CN 112784996 A CN112784996 A CN 112784996A
- Authority
- CN
- China
- Prior art keywords
- machine learning
- framework
- graph
- socket
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a machine learning method and system based on graph representation. Wherein the method comprises the following steps: the first equipment preprocesses the original data through the machine learning framework; the first device controls the machine learning framework to transmit the preprocessed processed data to the graph characterization framework through the first socket and the second socket; the second device performs graph characterization processing on the processed data through the graph characterization framework; the second device controls the graph characterization framework to transmit the graph characterization obtained by the graph characterization processing to the machine learning framework through the second socket and the first socket; the first device controls the machine learning framework to process an object to which the raw data belongs based on the graph representation. The performance of the algorithm model in machine learning can be improved.
Description
Technical Field
The invention relates to the technical field of machine learning, in particular to a machine learning method and system based on graph representation.
Background
In the machine learning, the algorithm model obtained through the machine learning may be processed according to the characterization of the object to be processed based on the mapping relationship between the learned characterization and the processing result, so as to obtain the processing result of the object to be processed, where the processing may be any one of classification processing, clustering processing, detection processing, regression processing, and the like. Therefore, the accuracy of the characterization will directly affect the performance of the algorithmic model.
Since objects often have some relationship between them, for example, one person may be a co-worker relationship with another person, one phone number may have called another phone number, a vehicle may be affiliated with a person, etc. Therefore, in the related art, in order to more accurately reflect the characteristics of the object, the graph characterization can be used as the characterization of the object, and the graph characterization can be calculated through graph characterization processing such as a graph embedding algorithm, a graph neural network algorithm and the like.
However, conventional machine learning frameworks often do not have graph representation processing capability, and therefore these machine learning frameworks cannot use graph representation as a representation of an object, resulting in poor performance of an algorithm model in machine learning.
Disclosure of Invention
The embodiment of the invention aims to provide a machine learning method and a machine learning system based on graph representation so as to improve the performance of an algorithm model in machine learning. The specific technical scheme is as follows:
in a first aspect of the embodiments of the present invention, a machine learning method based on graph representation is provided, which is applied to a machine learning system, where the machine learning system includes a first device and a second device, the first device is equipped with a machine learning frame, the second device is equipped with a graph representation frame, the machine learning frame is provided with a first socket, the graph representation frame is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the method comprises the following steps:
the first equipment preprocesses the original data through the machine learning framework;
the first device controls the machine learning framework to transmit the preprocessed processed data to the graph characterization framework through the first socket and the second socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework;
the second device controls the graph characterization framework to transmit the graph characterization obtained by the graph characterization processing to the machine learning framework through the second socket and the first socket;
the first device controls the machine learning framework to process an object to which the raw data belongs based on the graph representation.
In a possible embodiment, the second device controls the graph characterization framework to transmit the graph characterization processed by the graph characterization framework to the machine learning framework via the second socket and the first socket, and includes:
the second device controls the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real time via the second socket and the first socket.
In one possible embodiment, the first device controls the machine learning framework to process the object to which the raw data belongs based on the graph representation, including:
and the first equipment controls the machine learning framework to process the object to which the original data belongs in real time based on the received chart symptoms.
In one possible embodiment, the second device comprises a GPU, and the graph characterizes the framework running on the GPU of the second device.
In a possible embodiment, the first device controls the machine learning framework to transmit the preprocessed processed data to the graph characterization framework via the first socket and the second socket, and the method includes:
the first device controls the machine learning frame to perform format conversion on the processed data obtained through preprocessing to obtain first to-be-transmitted data in a preset third format, wherein the third format is a data format supported by the first socket and the second socket;
the first device controls the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework, and the graph characterization processing includes:
the second device controls the chart representation frame to perform format conversion on the first data to be transmitted to obtain chart representation input data in a preset second format, wherein the second format is a data format supported by the chart representation frame;
and the second equipment carries out graph characterization processing on the graph characterization input data through the graph characterization framework.
In a possible embodiment, the second device controls the graph characterization framework to transmit the graph characterization processed by the graph characterization framework to the machine learning framework via the second socket and the first socket, and includes:
the second equipment controls the chart feature frame to convert the format of the chart feature obtained by the chart feature processing, so that second to-be-transmitted data in a preset third format is obtained;
the second device controls the graph representation framework to send the second data to be transmitted to the first socket through the second socket;
the first device controls the machine learning framework to process the object to which the raw data belongs based on the graph representation, and the processing comprises the following steps:
the first device controls the machine learning frame to perform format conversion on the second data to be transmitted to obtain machine learning input data with a preset first format, wherein the first format is a data format supported by the machine learning frame;
the first device controls the machine learning framework to process an object to which the raw data belongs based on the machine learning input data.
In a second aspect of the embodiments of the present invention, a machine learning system based on graph representation is provided, where the machine learning system includes a first device and a second device, the first device is equipped with a machine learning framework, the second device is equipped with a graph representation framework, the machine learning framework is provided with a first socket, the graph representation framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the first device is used for preprocessing the original data through the machine learning framework; controlling the machine learning framework to transmit the processed data obtained through preprocessing to the graph characterization framework through the first socket and the second socket;
the second device is used for carrying out graph characterization processing on the processed data through the graph characterization framework; controlling the graph characterization framework to transmit the graph characterization obtained by the graph characterization processing to the machine learning framework through the second socket and the first socket;
the first device is further configured to control the machine learning framework to process an object to which the raw data belongs based on the graph representation.
In a possible embodiment, the second device is specifically configured to control the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real time via the second socket and the first socket.
In a possible embodiment, the first device is specifically configured to control the machine learning framework to process, in real time, an object to which the raw data belongs based on the received graph symptoms.
In one possible embodiment, the second device comprises a GPU, and the graph characterizes the framework running on the GPU of the second device.
In a possible embodiment, the first device is specifically configured to control the machine learning frame to perform format conversion on the processed data obtained through the preprocessing, so as to obtain first to-be-transmitted data in a preset third format, where the third format is a data format supported by the first socket and the second socket; controlling the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device is specifically configured to control the graph representation frame to perform format conversion on the first data to be transmitted to obtain graph representation input data in a preset second format, where the second format is a data format supported by the graph representation frame; and carrying out graph characterization processing on the graph characterization input data through the graph characterization framework.
In a possible embodiment, the second device is specifically configured to control the graph characterization framework to perform format conversion on a graph characterization obtained through graph characterization processing, so as to obtain second to-be-transmitted data in a preset third format; controlling the graph representation framework to send the second data to be transmitted to the first socket through the second socket;
the first device is specifically configured to control the machine learning frame to perform format conversion on the second data to be transmitted to obtain machine learning input data in a preset first format, where the first format is a data format supported by the machine learning frame; and controlling the machine learning framework to process the object to which the raw data belongs based on the machine learning input data.
The embodiment of the invention has the following beneficial effects:
according to the machine learning method and system based on the graph representation, provided by the embodiment of the invention, inter-program communication between the machine learning framework and the graph representation framework can be realized through sockets, so that data communication between the two frameworks is completed, and the graph representation output by the graph representation framework can be input into the machine learning framework, so that the machine learning framework without graph representation processing capacity can also perform machine learning based on the graph representation, and the performance of an algorithm model in machine learning is improved.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a machine learning method based on graph representation according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data flow provided by an embodiment of the present invention;
FIG. 3a is a timing diagram of a machine learning method based on graph representation according to an embodiment of the present invention;
FIG. 3b is a timing diagram illustrating another method for machine learning based on graph characterization according to an embodiment of the present invention;
FIG. 3c is a timing diagram illustrating another method for machine learning based on graph characterization according to an embodiment of the present invention;
FIG. 3d is another timing diagram illustrating a machine learning method based on graph characterization according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a machine learning system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to more clearly describe the machine learning method based on graph representation provided by the embodiment of the present invention, a possible application scenario of the machine learning method based on graph representation provided by the embodiment of the present invention will be described in the following by way of example, it can be understood that the following example is only one possible application scenario of the machine learning method based on graph representation provided by the embodiment of the present invention, and in other possible embodiments, the machine learning method based on graph representation provided by the embodiment of the present invention may also be applied to other possible application scenarios.
Suppose that a classification model for determining whether a person is a legal person needs to be trained for practical purposes. It is understood that although each of the legal person and the illegal person has some features, theoretically, the characteristics representing the feature of each person themselves can be used to distinguish the legal person from the illegal person, but the persons do not exist independently, for example, there may still exist person-to-person, person-to-vehicle, person-to-telephone number, and other association relationships between each person and other objects except the person, and these association relationships may also reflect to some extent whether the person is a legal person, for example, if there is an association relationship between one person and a plurality of legal persons, the person is a legal person with a high probability, and for example, if there is an association relationship between one person and an illegal IP address, the person is an illegal person with a high probability.
Therefore, the representation can effectively reflect the incidence relation between the person and other objects except the person, so that the trained model can be used for judging whether the person is a legal person or not by integrating the characteristics of multiple dimensions, and the accuracy of the judgment result is improved.
In order to enable the representation to effectively reflect the association relationship between the person and the other objects except the person, a relationship graph between each person and the other objects except the person may be constructed, and graph representation of the relationship graph is obtained by performing graph representation processing such as a graph embedding algorithm, a graph neural network algorithm and the like on the relationship graph, wherein the graph representation may be regarded as a vectorized representation of the relationship graph. The chart representation can effectively reflect the association relationship between the person and other objects except the person.
However, the conventional machine learning framework does not have Graph representation processing capability, and therefore, in a possible solution provided by the embodiment of the present invention, in order to enable the machine learning framework to perform machine learning based on Graph representation, a machine learning framework, such as a Spark (open source cluster computing environment) framework, and a Graph representation framework with Graph representation capability, such as a GNN (Graph Neural Networks) framework, may be separately constructed, and the machine learning framework and the Graph representation framework are respectively disposed in two different devices, hereinafter, for convenience of description, a device provided with the machine learning framework is referred to as a first device, and a device provided with the Graph representation framework is referred to as a second device.
The data representing the characteristics of each person and the association relationship between each person and other objects may be input to the first device, the first device performs preprocessing on the data by using a machine learning framework, such as preprocessing of extraction, cleaning, conversion, and the like, to obtain preprocessed data, and the first device stores the preprocessed data in a local storage system of the first device, where the machine learning framework is Spark, for example, the preprocessed data may be stored in a local distributed file system of the first device.
The first device sends the locally stored preprocessed data to the second device through a network between the first device and the second device, and the second device stores the received preprocessed data in a local storage system of the second device.
And the graph characterization framework in the second equipment reads the stored preprocessed data from the local storage system of the second equipment to perform graph characterization on the preprocessed data, so as to obtain graph characterizations which can be used for representing the self characteristics of each person and the association relationship between each person and other objects, and stores the obtained graph characterizations in the local storage system of the second equipment.
The second device sends the locally stored graph representation to the first device through a network between the second device and the first device, and the first device stores the received graph representation in a storage system local to the first device. And a machine learning framework in the first equipment reads the stored graph characteristics from a local storage system of the first equipment, performs machine learning based on the read graph characteristics, and trains to obtain a classification model for judging whether the personnel is legal or not.
However, in this scheme, it takes a certain time for data reading and writing between the machine learning frame and the local storage system of the first device, data reading and writing between the graph characterization frame and the local storage system of the second device, and data interaction between the first device and the second device through the network, which results in low machine learning efficiency.
Based on this, an embodiment of the present invention provides a machine learning method based on graph representation, which may be referred to fig. 1, where fig. 1 is a schematic flow diagram of the machine learning method based on graph representation provided in the embodiment of the present invention, and the method may include:
s101, the first device preprocesses the original data through a machine learning framework.
S102, the first equipment control machine learning framework transmits the processed data obtained through preprocessing to the graph characterization framework through the first socket and the second socket.
And S103, the second device performs graph characterization processing on the processed data through a graph characterization framework.
And S104, the second device control diagram representation framework transmits the graph representation obtained through the graph representation processing to the machine learning framework through the second socket and the first socket.
And S105, the first equipment control machine learning framework processes the object to which the original data belongs based on the graph representation.
By adopting the embodiment, the inter-program communication between the technical machine learning framework and the chart characterization framework can be realized through the socket, so that the data communication between the two frameworks is completed, the chart characterization output by the chart characterization framework can be input into the machine learning framework, the machine learning framework without the chart characterization processing capacity can also perform machine learning based on the chart characterization, and the performance of the algorithm model in the machine learning is improved.
In S101, the first device is a device equipped with a machine learning framework, the first device may be one device or formed by integrating a plurality of devices, and the first device may be a physical device or a virtual device, which is not limited in this example.
The pre-processing may include one or more of extraction processing, cleaning processing, and conversion processing, and other processing may also be included in the pre-processing according to different application scenarios, which is not limited in this embodiment.
The raw data may be data representing the characteristics of each person and the association relationship between each person and other objects, for example, taking the application scenario as an example, or may be data representing the characteristics of each vehicle and the association relationship between each vehicle and other objects, for example, assuming that a model for classifying vehicles needs to be trained. The object in the present text may include one or more of a person, a vehicle, a phone number, a MAC address, an IMSI number, a location, a face image, a license plate number, and the like.
In S102, a graph characterization framework is disposed in the second device, the first socket is disposed in the machine learning framework, and the second socket is disposed in the graph characterization framework. The first socket and the second socket support the same data format and are interfaced, i.e., the machine learning framework and the graph characterization framework can enable inter-program communication through the first socket and the second socket.
The first socket and the second socket support the same data format means that the same data format is agreed to be used for data transmission in the transmission protocols of the first socket and the second socket, and for convenience of description, the data format supported by the machine learning framework is referred to as a first data format, the data format supported by the graph representation framework is referred to as a second data format, and the data format supported by the first socket and the second socket is referred to as a third data format.
In one possible implementation, to increase the transmission rate, the processed data may be compressed, and the compressed processed data may be transmitted to the graph characterization framework via the first socket and the second socket.
In S103, the graph characterization processing may be processing by using a graph embedding algorithm or a graph neural network algorithm, and the type of the graph characterization processing should be the type of the graph characterization processing supported by the graph characterization framework. Illustratively, if the graph characterizing framework is a knowledge graph embedding framework, the graph characterizing process is a process using a graph embedding algorithm, and if the graph characterizing framework is a graph neural network framework, the graph characterizing process is a process using a graph neural network algorithm. Since the graph representation process is not a main improvement point of the present invention, the detailed flow of the graph representation process is not described herein.
In one possible embodiment, to improve the efficiency of the graph characterization framework in performing the graph characterization process, the second device may include a GPU in which the graph characterization framework runs. By adopting the embodiment, the strong graphic operation capability of the GPU can be utilized, and the efficiency of the graphic representation processing of the graphic representation framework is effectively improved.
In S104, in order to increase the transmission rate in one possible implementation, the graph characterization may be compressed, and the compressed graph characterization may be transmitted to the machine learning framework via the second socket and the first socket.
In S105, the object to which the raw data belongs may refer to a collection object when the raw data is collected, and for example, if the raw data is obtained by collecting each person, the object to which the raw data belongs is a person.
If the original data is sample data obtained by collecting a sample object, a processing result obtained by processing the object to which the original data belongs can be used for model training, and if the original data is data obtained by collecting an object to be processed, the processing result obtained by processing the object to which the original data belongs is the processing result of the object to be processed.
The machine learning method based on graph representation provided by the embodiment of the invention can be applied to a scene of an algorithm model training, and can also be applied to a scene of processing an object by using the algorithm model after the algorithm model is obtained through training.
The processing herein may refer to classification processing, detection processing, clustering processing, regression processing, and other processing besides classification, detection, clustering, and regression, which is not limited in this embodiment.
For a clearer description of the machine learning method based on graph representation provided by the embodiment of the present invention, refer to fig. 2, and fig. 2 is a schematic diagram illustrating a data flow direction provided by the embodiment of the present invention, where a thin solid line with an arrow indicates the data flow direction in the machine learning method based on graph representation provided by the embodiment of the present invention, and a thick solid line with an arrow indicates the data flow direction in the machine learning method described in the foregoing application scenario.
Therefore, in the machine learning method based on graph representation provided by the embodiment of the invention, direct communication between the machine learning frame and the graph representation frame is realized through the socket, and reading and writing between the machine learning frame and the storage system of the first device and reading and writing between the graph representation frame and the storage system of the second device are omitted, so that the machine learning efficiency can be effectively improved, and the landing of an algorithm model in machine learning is accelerated.
In a possible embodiment, the five steps S101-S105 may be executed in series, i.e. S102 is executed after S101 is completed, S103 is executed after S102 is completed, and so on, and the timing diagram of these five steps may be as shown in fig. 3 a.
Assuming that the time duration consumed by S101 is t1, the time duration consumed by S102 is t2, the time duration consumed by S103 is t3, the time duration consumed by S104 is t4, and the time duration consumed by S105 is t5, the total time duration consumed by executing the graph representation-based machine learning method provided by the embodiment of the invention according to the timing diagram shown in fig. 3a is t1+ t2+ t3+ t4+ t 5.
In some possible embodiments, the total time consumed for executing the graph representation-based machine learning method provided by the embodiment of the present invention may be reduced by a parallel execution manner. It can be understood that, when the graph representation processing is performed, in order to enable the obtained graph representation to represent the association relationship between the object and other objects as comprehensively as possible, the graph representation processing needs to be performed based on the completely processed data, that is, S103 needs to be performed after the execution of S102 is completed.
The graph representation is represented in the form of a vector, the values of the elements in the vector are determined at different times, i.e. the time period from the first element of the vector to the last element of the vector is up to, and the execution of S104 can be started from the first element of the vector. Thus, in one possible embodiment, the second device may be a control chart characterization framework that transmits the generated chart characterization to the machine learning framework via the second socket and the first socket in real time.
At this time, a timing chart of the graph representation-based machine learning method provided by the embodiment of the present invention may be as shown in fig. 3 b. As shown in fig. 3b, it can be seen that the total time consumed by executing the graph representation-based machine learning method provided by the embodiment of the present invention according to the timing chart shown in fig. 3b is significantly less than t1+ t2+ t3+ t4+ t5, that is, the machine learning efficiency can be effectively improved by using this embodiment.
It will be appreciated that it may not be necessary to characterize all features based on the graph when processing is performed, and therefore in one possible embodiment, S105 may already be performed when the machine learning framework receives the graph characterization, i.e. the first device may control the machine learning framework to process the object to which the raw data belongs in real time based on the received graph characterization.
At this time, a timing chart of the graph representation-based machine learning method provided by the embodiment of the present invention may be as shown in fig. 3 c. As shown in fig. 3c, it can be seen that the total time consumed by executing the graph representation-based machine learning method provided by the embodiment of the present invention according to the timing chart shown in fig. 3c is significantly less than t1+ t2+ t3+ t4+ t5, that is, the machine learning efficiency can be effectively improved by using this embodiment.
In yet another possible embodiment, the second device may be a control chart representation framework that transmits the generated chart representation to the machine learning framework in real time via the second socket and the first socket, and the first device may control the machine learning framework to process the object to which the raw data belongs in real time based on the received chart representation.
At this time, a timing chart of the graph representation-based machine learning method provided by the embodiment of the present invention may be as shown in fig. 3 d. As shown in fig. 3d, it can be seen that the total time consumed by executing the graph representation-based machine learning method provided by the embodiment of the present invention according to the timing chart shown in fig. 3d is significantly less than t1+ t2+ t3+ t4+ t5, that is, the machine learning efficiency can be effectively improved by using this embodiment.
It will be appreciated that the first data format supported by the machine learning framework and the second data format supported by the graph characterization framework tend to be different.
Therefore, when the machine learning framework transmits the processed data to the graph characterization framework, the first device may control the machine learning framework to perform format conversion on the pre-processed data, so as to obtain the first to-be-transmitted data in the third format. And controlling, by the first device, the machine learning framework to send the first data to be transmitted to the second socket via the first socket.
After the graph characterization framework receives the first to-be-transmitted data through the second socket, the second device control graph characterization framework can perform format conversion on the first to-be-transmitted data to obtain graph characterization input data in a second format, and the graph characterization input data is subjected to graph characterization processing through the graph characterization framework. It is understood that the graph characterization input data is obtained by format conversion of the first data to be transmitted, and the first data to be transmitted is obtained by format conversion of the processed data, so that the graph characterization processing performed on the graph characterization input data can be regarded as performing the graph characterization processing on the processed data.
Similarly, when the graph characteristic frame transmits the graph characteristic data to the machine learning frame, the second device may control the machine learning frame to perform format conversion on the graph characteristic obtained through the graph characteristic processing, so as to obtain second to-be-transmitted data in a third format. And sending the second data to be transmitted to the first socket through the second socket by the second device control map representation framework.
After the machine learning frame receives the second data to be transmitted through the first socket, the first device can control the machine learning frame to perform format conversion on the second data to be transmitted to obtain machine learning input data in the first format, and the machine learning frame processes the object to which the original data belongs based on the machine learning input data. It can be understood that the machine learning input data is obtained by format conversion of the second data to be transmitted, and the second data to be transmitted is obtained by format conversion of the graph representation, so that the object to which the original data belongs is processed based on the machine learning input data, and the object to which the original data belongs can be regarded as being processed based on the graph representation.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a machine learning system based on graph representation according to an embodiment of the present invention, where the machine learning system includes a first device 410 and a second device 420, the first device is equipped with a machine learning frame 411, the second device is equipped with a graph representation frame 421, the machine learning frame is provided with a first socket, the graph representation frame is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the first device 410, configured to perform preprocessing on raw data through the machine learning framework 411; controlling the machine learning framework 411 to transmit the preprocessed data to the graph representation framework 421 via the first socket and the second socket;
the second device 420 is configured to perform graph characterization processing on the processed data through the graph characterization framework 421; controlling the graph representation framework 421 to transmit the graph representation processed by the graph representation processing to the machine learning framework 411 through the second socket and the first socket;
the first device 410 is further configured to control the machine learning framework 411 to process an object to which the raw data belongs based on the graph representation.
In a possible embodiment, the second device 420 is specifically configured to control the graph characterization framework 421 to transmit the generated graph characterization to the machine learning framework 411 in real time via the second socket and the first socket.
In a possible embodiment, the first device 410 is specifically configured to control the machine learning framework 411 to process, in real time, an object to which the raw data belongs based on the received graph characteristics.
In one possible embodiment, the second device 420 comprises a GPU, and the graph characterizes the framework 421 running on the GPU of the second device.
In a possible embodiment, the first device 410 is specifically configured to control the machine learning framework 411 to perform format conversion on the processed data obtained through the preprocessing, so as to obtain first to-be-transmitted data in a preset third format, where the third format is a data format supported by the first socket and the second socket; control the machine learning framework 411 to send the first data to be transmitted to the second socket through the first socket;
the second device 420 is specifically configured to control the graph representation frame 421 to perform format conversion on the first data to be transmitted to obtain graph representation input data in a preset second format, where the second format is a data format supported by the graph representation frame 421; graph characterization processing is performed on the graph characterization input data by the graph characterization framework 421.
In a possible embodiment, the second device 420 is specifically configured to control the graph characterization frame 421 to perform format conversion on a graph characterization obtained by the graph characterization processing, so as to obtain second to-be-transmitted data in a preset third format; controlling the graph representation framework 421 to send the second data to be transmitted to the first socket through the second socket;
the first device 410 is specifically configured to control the machine learning framework 411 to perform format conversion on the second data to be transmitted, so as to obtain machine learning input data in a preset first format, where the first format is a data format supported by the machine learning framework 411; controlling the machine learning framework 411 to process the object to which the raw data belongs based on the machine learning input data.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. The machine learning method based on graph representation is applied to a machine learning system, the machine learning system comprises a first device and a second device, the first device is provided with a machine learning framework, the second device is provided with a graph representation framework, the machine learning framework is provided with a first socket, the graph representation framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the method comprises the following steps:
the first equipment preprocesses the original data through the machine learning framework;
the first device controls the machine learning framework to transmit the preprocessed processed data to the graph characterization framework through the first socket and the second socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework;
the second device controls the graph characterization framework to transmit the graph characterization obtained by the graph characterization processing to the machine learning framework through the second socket and the first socket;
the first device controls the machine learning framework to process an object to which the raw data belongs based on the graph representation.
2. The method of claim 1, wherein the second device controls the graph characterization framework to transmit the graph characterization processed by the graph characterization framework to the machine learning framework via the second socket and the first socket, and comprises:
the second device controls the graph characterization framework to transmit the generated graph characterization to the machine learning framework in real time via the second socket and the first socket.
3. The method of claim 2, wherein the first device controls the machine learning framework to process the object to which the raw data belongs based on the graph representation, comprising:
and the first equipment controls the machine learning framework to process the object to which the original data belongs in real time based on the received chart symptoms.
4. The method of claim 1, wherein the second device comprises a GPU, and wherein the graph characterizes a framework running on the GPU of the second device.
5. The method of claim 1, wherein the first device controlling the machine learning framework to transmit the pre-processed data to the graph characterization framework via the first socket and the second socket comprises:
the first device controls the machine learning frame to perform format conversion on the processed data obtained through preprocessing to obtain first to-be-transmitted data in a preset third format, wherein the third format is a data format supported by the first socket and the second socket;
the first device controls the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device performs graph characterization processing on the processed data through the graph characterization framework, and the graph characterization processing includes:
the second device controls the chart representation frame to perform format conversion on the first data to be transmitted to obtain chart representation input data in a preset second format, wherein the second format is a data format supported by the chart representation frame;
and the second equipment carries out graph characterization processing on the graph characterization input data through the graph characterization framework.
6. The method of claim 1, wherein the second device controls the graph characterization framework to transmit the graph characterization processed by the graph characterization framework to the machine learning framework via the second socket and the first socket, and comprises:
the second equipment controls the chart feature frame to convert the format of the chart feature obtained by the chart feature processing, so that second to-be-transmitted data in a preset third format is obtained;
the second device controls the graph representation framework to send the second data to be transmitted to the first socket through the second socket;
the first device controls the machine learning framework to process the object to which the raw data belongs based on the graph representation, and the processing comprises the following steps:
the first device controls the machine learning frame to perform format conversion on the second data to be transmitted to obtain machine learning input data with a preset first format, wherein the first format is a data format supported by the machine learning frame;
the first device controls the machine learning framework to process an object to which the raw data belongs based on the machine learning input data.
7. The machine learning system based on graph representation is characterized by comprising a first device and a second device, wherein the first device is provided with a machine learning framework, the second device is provided with a graph representation framework, the machine learning framework is provided with a first socket, the graph representation framework is provided with a second socket, and the first socket and the second socket support the same data format and are in butt joint;
the first device is used for preprocessing the original data through the machine learning framework; controlling the machine learning framework to transmit the processed data obtained through preprocessing to the graph characterization framework through the first socket and the second socket;
the second device is used for carrying out graph characterization processing on the processed data through the graph characterization framework; controlling the graph characterization framework to transmit the graph characterization obtained by the graph characterization processing to the machine learning framework through the second socket and the first socket;
the first device is further configured to control the machine learning framework to process an object to which the raw data belongs based on the graph representation.
8. The system of claim 7, wherein the second device is specifically configured to control the graph characterization framework to transmit the generated graph characterization to the machine learning framework via the second socket and the first socket in real time.
9. The system of claim 8, wherein the first device is specifically configured to control the machine learning framework to process the object to which the raw data belongs in real-time based on the received graphical representation.
10. The system of claim 7, wherein the second device comprises a GPU and the graph characterization framework runs on the GPU of the second device.
11. The system according to claim 7, wherein the first device is specifically configured to control the machine learning framework to perform format conversion on the processed data obtained through the preprocessing, so as to obtain first data to be transmitted in a preset third format, where the third format is a data format supported by the first socket and the second socket; controlling the machine learning framework to send the first data to be transmitted to the second socket through the first socket;
the second device is specifically configured to control the graph representation frame to perform format conversion on the first data to be transmitted to obtain graph representation input data in a preset second format, where the second format is a data format supported by the graph representation frame; and carrying out graph characterization processing on the graph characterization input data through the graph characterization framework.
12. The system according to claim 7, wherein the second device is specifically configured to control the graph characterization framework to perform format conversion on a graph characterization obtained by the graph characterization processing, so as to obtain second to-be-transmitted data in a preset third format; controlling the graph representation framework to send the second data to be transmitted to the first socket through the second socket;
the first device is specifically configured to control the machine learning frame to perform format conversion on the second data to be transmitted to obtain machine learning input data in a preset first format, where the first format is a data format supported by the machine learning frame; and controlling the machine learning framework to process the object to which the raw data belongs based on the machine learning input data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637601.5A CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637601.5A CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112784996A true CN112784996A (en) | 2021-05-11 |
| CN112784996B CN112784996B (en) | 2023-06-02 |
Family
ID=75754984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011637601.5A Active CN112784996B (en) | 2020-12-31 | 2020-12-31 | Machine learning method and system based on graph representation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112784996B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2870571A2 (en) * | 2012-07-09 | 2015-05-13 | Toyota Motor Europe NV/SA | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| US20150178620A1 (en) * | 2011-07-07 | 2015-06-25 | Toyota Motor Europe Nv/Sa | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| US20190073586A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Nested Machine Learning Architecture |
| US20190073581A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Mixed Machine Learning Architecture |
| US20190073580A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Sparse Neural Network Modeling Infrastructure |
| US20190287032A1 (en) * | 2018-03-16 | 2019-09-19 | International Business Machines Corporation | Contextual Intelligence for Unified Data Governance |
| CN111245673A (en) * | 2019-12-30 | 2020-06-05 | 浙江工商大学 | SDN time delay sensing method based on graph neural network |
| CN111899150A (en) * | 2020-08-28 | 2020-11-06 | Oppo广东移动通信有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111914180A (en) * | 2020-08-19 | 2020-11-10 | 腾讯科技（深圳）有限公司 | User characteristic determination method, device, equipment and medium based on graph structure |
| CN111930518A (en) * | 2020-09-22 | 2020-11-13 | 北京东方通科技股份有限公司 | Knowledge graph representation learning-oriented distributed framework construction method |
| CN112035261A (en) * | 2020-09-11 | 2020-12-04 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
| CN112069398A (en) * | 2020-08-24 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information pushing method and device based on graph network |
| CN112069412A (en) * | 2020-09-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information recommendation method and device, computer equipment and storage medium |
-
2020
- 2020-12-31 CN CN202011637601.5A patent/CN112784996B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150178620A1 (en) * | 2011-07-07 | 2015-06-25 | Toyota Motor Europe Nv/Sa | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| EP2870571A2 (en) * | 2012-07-09 | 2015-05-13 | Toyota Motor Europe NV/SA | Artificial memory system and method for use with a computational machine for interacting with dynamic behaviours |
| US20190073586A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Nested Machine Learning Architecture |
| US20190073581A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Mixed Machine Learning Architecture |
| US20190073580A1 (en) * | 2017-09-01 | 2019-03-07 | Facebook, Inc. | Sparse Neural Network Modeling Infrastructure |
| US20190287032A1 (en) * | 2018-03-16 | 2019-09-19 | International Business Machines Corporation | Contextual Intelligence for Unified Data Governance |
| CN111245673A (en) * | 2019-12-30 | 2020-06-05 | 浙江工商大学 | SDN time delay sensing method based on graph neural network |
| CN111914180A (en) * | 2020-08-19 | 2020-11-10 | 腾讯科技（深圳）有限公司 | User characteristic determination method, device, equipment and medium based on graph structure |
| CN112069398A (en) * | 2020-08-24 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information pushing method and device based on graph network |
| CN111899150A (en) * | 2020-08-28 | 2020-11-06 | Oppo广东移动通信有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112035261A (en) * | 2020-09-11 | 2020-12-04 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
| CN112069412A (en) * | 2020-09-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Information recommendation method and device, computer equipment and storage medium |
| CN111930518A (en) * | 2020-09-22 | 2020-11-13 | 北京东方通科技股份有限公司 | Knowledge graph representation learning-oriented distributed framework construction method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112784996B (en) | 2023-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109639481B (en) | Deep learning-based network traffic classification method and system and electronic equipment | |
| US10467490B2 (en) | User identity verification method, apparatus and system | |
| US11646972B2 (en) | Dynamic allocation of network resources using external inputs | |
| US9043076B2 (en) | Automating predictive maintenance for automobiles | |
| CN110138745B (en) | Abnormal host detection method, device, equipment and medium based on data stream sequence | |
| JP2019513246A (en) | Training method of random forest model, electronic device and storage medium | |
| CN105740707A (en) | Malicious file identification method and device | |
| CN111191767A (en) | Vectorization-based malicious traffic attack type judgment method | |
| CN107528904B (en) | Method and apparatus for data distributed anomaly detection | |
| CN110211121B (en) | Method and device for pushing model | |
| CN113037783A (en) | Abnormal behavior detection method and system | |
| US8712100B2 (en) | Profiling activity through video surveillance | |
| WO2017054307A1 (en) | Recognition method and apparatus for user information | |
| CN109614780B (en) | Biological information authentication method and device, storage medium and electronic equipment | |
| US10600067B2 (en) | Demographic based adjustment of data processing decision results | |
| CN111444364B (en) | Image detection method and device | |
| CN113746780A (en) | Abnormal host detection method, device, medium and equipment based on host image | |
| CN112784996A (en) | Machine learning method and system based on graph representation | |
| CN111737371B (en) | Data flow detection classification method and device capable of dynamically predicting | |
| CN115049843A (en) | Confrontation sample generation method and device, electronic equipment and storage medium | |
| CN112486676B (en) | Data sharing and distributing device based on edge calculation | |
| CN110414845B (en) | Risk assessment method and device for target transaction | |
| CN117097578B (en) | Network traffic safety monitoring method, system, medium and electronic equipment | |
| CN117478685B (en) | Group control device identification method, device, equipment and readable storage medium | |
| CN110365675B (en) | Method, device and system for network tracking long chain attack |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |