CN108510082B - Method and device for processing machine learning model - Google Patents
Method and device for processing machine learning model Download PDFInfo
- Publication number
- CN108510082B CN108510082B CN201810258874.5A CN201810258874A CN108510082B CN 108510082 B CN108510082 B CN 108510082B CN 201810258874 A CN201810258874 A CN 201810258874A CN 108510082 B CN108510082 B CN 108510082B
- Authority
- CN
- China
- Prior art keywords
- model
- experiment
- prediction
- machine learning
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
Abstract
The embodiment of the invention discloses a method and a device for processing a machine learning model, relates to the technical field of Internet, and can improve the real-time performance of services based on the machine learning model. The invention comprises the following steps: running a training experiment through a machine learning platform, generating a model file corresponding to the training experiment, and creating a prediction experiment according to the model file; acquiring, by the machine learning platform, experimental data corresponding to the prediction experiment, the experimental data including: XML information, an algorithm package on which the prediction experiment depends; and calling an algorithm package depended by the prediction experiment to analyze the XML information, and executing the prediction experiment after the analysis result is loaded to the model. The method is suitable for machine learning model services with strong real-time requirements.
Description
Technical Field
The invention relates to the technical field of Internet e-commerce, in particular to a method and a device for processing a machine learning model.
Background
With the development of internet technology, machine learning technology is gradually applied to various online services, and in the most common field at present, a corresponding response robot is established through a machine learning training model, so that the automation degree of the online services is improved, and the operation efficiency of a service system is improved. Currently, in the training and deployment process of a machine learning model, the traditional practice in the industry is as follows: the model is stored into a pmml format file, and then the pmml format file is analyzed by using a tool (such as JPMML) corresponding to the pmml format file or a custom analysis method, and then an experiment is carried out. However, the pmml mode is required, the algorithm code is required to meet specific specifications by the pmml mode, the rules are complex, the code writing is complicated, the code needs to be written according to a specific format/specification, some self-defined algorithms are difficult to write, time and labor are wasted, the model development efficiency is low, and the model is difficult to adapt to the service system with higher real-time requirement.
Disclosure of Invention
The embodiment of the invention provides a method and a device for processing a machine learning model, which can improve the development efficiency based on the machine learning model.
In order to achieve the above purpose, the embodiment of the invention adopts the following technical scheme:
running a training experiment through a machine learning platform, generating a model file corresponding to the training experiment, and creating a prediction experiment according to the model file; acquiring, by the machine learning platform, experimental data corresponding to the prediction experiment, the experimental data including: XML information, the algorithm package on which the prediction experiment depends; and calling an algorithm package depended by the prediction experiment to analyze the XML information, and executing the prediction experiment after the analysis result is loaded to the model.
Through the machine learning platform in the embodiment, automatic training for establishing a model of the robot is realized, especially, experimental preparation of the model is realized through training experiments, relevant experimental data are automatically extracted, and then the training effect of the model is automatically verified through prediction experiments. The preparation and training process of the model file is not limited by the pmml format any more, so that the development efficiency of the model is improved, and the model can better adapt to a service system with higher real-time requirement.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1a and fig. 1b are schematic diagrams of a providing architecture according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a method provided by an embodiment of the present invention;
fig. 3, 4 and 5 are schematic diagrams of specific examples provided by the embodiment of the invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the present invention will be described in further detail with reference to the accompanying drawings and specific embodiments. Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention. As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The method flow in this embodiment may be specifically implemented on a system as shown in fig. 1a, and specifically, the system includes:
the machine learning platform, the off-line computing platform and the Spark computing cluster, and the end devices of the system can establish channels through the internet and perform data interaction through respective data transmission ports.
The off-line computing platform disclosed in this embodiment may be specifically a workstation, a super computer, or a server set for data processing, which is composed of multiple servers. The off-line computing platform is specifically used for acquiring XML information, a dependent algorithm package and a model file required by an experiment, and submitting an execution instruction to spark.
Spark computing cluster, including a plurality of computing nodes, each computing node can be a server, a computer or a virtual machine established by the allocated computing resources, a Spark process executing tasks as a client applies for resources (computation nodes) from the "cluster", after the "cluster" allocates resources, the Spark process can decompose some computing work and put them into the applied resources to run.
The machine learning platform may be specifically composed of a plurality of servers, hardware devices having a calculation function such as supercomputing devices, and the like on a hardware level.
An embodiment of the present invention provides a method for processing a machine learning model, as shown in fig. 2, including:
s1, running a training experiment through a machine learning platform, generating a model file corresponding to the training experiment, and creating a prediction experiment according to the model file.
It should be noted that, the platform for training the machine learning-based model is generally referred to as a machine learning platform in the industry. The "experiment" may be understood as a generic term for one or more tasks performed from a single entry point, and may include tasks such as data introduction, feature engineering, model training, or model evaluation. Among them, an experiment targeting a training model is referred to as a training experiment. The experiment in which the trained model is used as a task node in the experiment is called a prediction experiment.
The model file is specifically understood to be a file in which a model for an experiment is stored in a computer, and the model is a serialized binary document. And (4) establishing a training experiment on the machine learning platform, and generating a corresponding model file after the experiment is run. In the prediction process, the model file can be deserialized into a model, and the model parent has a prediction interface. And then, according to the generated model, a corresponding prediction experiment is created again on the machine learning platform.
Herein, "parent class of model" can be understood in the art-recognized meaning, that is, the parent class of the model refers to the base class of all algorithmic model classes, and the parent class of the model defines interfaces commonly required by the model, such as fit and transform interfaces.
And S2, acquiring experiment data corresponding to the prediction experiment through the machine learning platform.
Wherein the experimental data comprises: XML information, the algorithm package on which the prediction experiment depends. The off-line computing platform can obtain XML information, a dependent algorithm package and a model file of the prediction experiment through an interface provided by the machine learning platform.
Specifically, the XML information includes: and predicting the step flow of the experiment execution, wherein each step corresponds to one jobb task, each jobb task comprises the logic class and parameter information of the corresponding step, and the output result of the previous jobb task is used as the input of the next jobb task. For example: the XML information contains the order and steps in which the experiment is performed. Each step is taken as an independent job task, and each job task comprises the processing logic class of the current stage and relevant parameters. The output result of the previous job task will be used as the input of the next job task.
And S3, calling an algorithm package depended by the prediction experiment to analyze the XML information, and executing the prediction experiment after the analysis result is loaded to the model.
Through the machine learning platform in the embodiment, automatic training for establishing a model of the robot is realized, especially, experimental preparation of the model is realized through training experiments, relevant experimental data are automatically extracted, and then the training effect of the model is automatically verified through prediction experiments. The preparation and training process of the model file is not limited by the pmml format any more, so that the development efficiency of the model is improved, and the model can better adapt to a service system with higher and higher real-time requirement.
Specifically, configuration information of all nodes in the experiment is stored in an xml format, and the nodes include but are not limited to: the XML information specifically comprises information such as parameters of each node and dependency among the nodes.
Specifically, the algorithm package includes: the source code of each task node in the prediction experiment includes: and various algorithms such as data preprocessing, feature engineering, regression classification clustering time sequence and the like.
The method comprises the steps of running a training experiment through a machine learning platform, generating a model file for establishing the robot, and establishing a prediction experiment according to the model file to check the accuracy of the model file. And obtaining a model if the model file is deserialized, wherein the parent class of the model is provided with an interface corresponding to the prediction experiment. Meanwhile, the machine learning platform also reads the XML information and the algorithm package, and analyzes the XML information through the algorithm package, so that the parameters needing to be imported into the model to operate are obtained, the model can normally operate, and the prediction experiment is completed.
Further, the method also comprises the following steps: and when the model passes the prediction experiment, releasing the model, wherein the model is used for establishing the robot in a target business system. The machine learning platform can automatically or manually select by a user to publish the model to the target business system so that the target business system can build the robot according to the published model. Specifically, the operation and maintenance personnel of the target business system can establish the robot for the target business system through the published model.
The model can be automatically released to the target business system, so that the model deployment efficiency is improved, and the real-time performance of the model development and release process is further improved. Therefore, through the machine learning platform, efficient development and efficient release of the model for establishing the robot are achieved.
For example: when the embodiment is applied to the scenes of off-line prediction and quasi-real-time prediction, the method can be expanded to the flow shown in fig. 3:
s101: a training experiment is created. Specifically, a training experiment is established by the machine learning platform and used for cattle prevention detection, and random forest secondary classification is used for algorithm nodes.
S102: training the experiment and generating the model. Specifically, after the experiment training is finished, the random forest classification nodes generate corresponding models, the models are serialized binary files, a predicted function is arranged in the binary files, and the prediction can be achieved by calling the function.
S200: a prediction experiment was created. Specifically, the machine learning platform applies the model to create an experiment for predicting whether it is a cattle.
S300: and acquiring XML information, a dependent algorithm package and a model file of the experiment. Specifically, the off-line computing platform acquires the XML information of the test through a machine learning interface, and replaces and maps the data source and the real data source. Wherein, if the scheme is an off-line prediction scheme, the real data source is the HIVE table of the service. In the case of the quasi-real-time scheme, the real data source is an MQ (Message Queue), so that configuration information of the MQ is mapped, and data to be predicted can be connected and acquired through the configuration information.
S401: the execute instruction is committed to spark. Specifically, the offline computing platform submits an instruction for executing the prediction experiment to the spark in a spark-submit mode, and the instruction specifies XML information, an algorithm package, a model file, an operating parameter of the experiment, information such as a class full path where a main function is located, and the like.
S402: and analyzing XML information of the experiment to find out a jobtask input by the active table. Specifically, the Spark runs the class of the main function, analyzes the XML information of the experiment, and finds the job task input by the active table. S403: and according to the mapping relation between the source table and the data source, taking out data from the corresponding HIVE (the HIVE is slow and is used for offline) or MQ (the MQ is fast and is used for quasi-real time) and converting the data into RDDs (flexible Distributed data sets). Specifically, according to the mapping relationship between the source table and the data source, the corresponding HIVE table or MQ is found, the data is read, and the data is converted into the RDD format. According to a specific application scene, the data source can be flexibly adjusted by switching the interface, so that the embodiment can meet different scenes such as off-line, quasi-real-time and real-time scenes.
S404: and judging whether the job task is of a model prediction type or other types. Specifically, the job task information in the experimental XML information is sequentially executed downwards. In the process, whether the job task is a model prediction node of random forest secondary classification or other nodes (such as data preprocessing nodes: type conversion/missing value filling and the like) needs to be judged.
S405: if the model is the prediction type, the corresponding model file is deserialized, then the prediction method is called, and the result is output to the intermediate table. Specifically, if the job task is of a random forest two-class type, the model file name (such as a 'cattle prediction model 1') is analyzed according to the information in the job task, then the model file is loaded into a memory according to the uploaded model file, deserialized, and then the prediction function of the model file is called to perform operation, and the result is written into an intermediate table (if the node is the last node, the result is written into a final result HIVE table or a corresponding MQ).
S406: if the type is other, executing the corresponding algorithm and outputting the result to the intermediate table. Specifically, if the job task is of another operation type, such as a String type converted into a Double type, the corresponding algorithm function in the algorithm package is directly loaded and called for processing. The results of the processing are written as above in the intermediate table/final results table/MQ.
S407: and sequentially executing all job tasks downwards until the job tasks are finished. The final result falls into the HIVE table or MQ of the target.
S500: the off-line computing platform in the above steps is scheduled periodically and circularly. Specifically, the offline computing platform periodically triggers scheduling every 5 to 10 minutes.
In the current mainstream training and deployment scheme of the machine learning model: the model is firstly saved into PMML format file, and then the PMML format file is analyzed by JPMML tool or self-defined analysis method to predict. However, the pmml mode is required, and the mode requires that the algorithm code must meet a specific specification, the rule is complex, so that the code writing is complicated, the code writing is required to be performed according to a specific format/specification, and some self-defined algorithms are difficult to write. The method is time-consuming and labor-consuming, and is difficult to further meet the real-time requirement of the current online service.
In this embodiment, according to different real-time level requirements, an architecture scheme configured with the method can be adopted, so that the complexity of codes and deployment is greatly reduced, and one-key deployment and calling are realized. And the complexity of coding is greatly reduced, the algorithm logic of the self is freely customized, and the requirement of meeting a specific standard writing algorithm is not required (the traditional technical scheme needs to write according to a certain standard so as to generate a pmml format file), so that the freedom of the code is realized. The whole experiment or model can complete automatic deployment and can cover various scenes such as off-line, quasi-real-time and real-time.
Furthermore, according to the mapping relation between the source table and the data source, data is taken out from the corresponding HIVE (the HIVE is slow and used for offline) or MQ (the MQ is fast and used for quasi-real-time) and converted into RDD. Specifically, according to the mapping relationship between the source table and the data source, the corresponding HIVE table or MQ is found, the data is read, and the data is converted into the RDD format. According to a specific application scene, the data source can be flexibly adjusted by switching the interface, so that the embodiment can meet different scenes such as off-line, quasi-real-time, real-time and the like.
The embodiment is applied to two scenes of off-line prediction and quasi-real-time prediction, has the capability of processing large data volume, and is very suitable for scenes with large user volume; according to the real-time prediction scheme, unified allocation and scheduling of resources can be achieved according to application scenarios of users, so that parallel expansion is facilitated, and the method and the device are particularly suitable for scenarios with seasonal, holiday and sudden access volume surge.
In this embodiment, after the step S3 of loading the analysis result into the model, a specific implementation manner of executing the prediction experiment may include:
and calling an algorithm package depended by the prediction experiment, extracting an active table from the XML information, and inputting a jobtask 1. And extracting source data according to the mapping relation between the active table and the data source, converting the source data into RDD (resource description data), executing the job task 1 according to the RDD obtained by conversion, and outputting the operation result of the job task 1 to a target table.
In particular, an algorithm package may be understood as a collection of codes for algorithms supported by a machine learning platform. In this embodiment, the algorithm package is shared by the machine learning platform and the model service platform, and the algorithm package may be stored in the machine learning platform, the model service platform, or a storage device connected to the machine learning platform and the model service platform. The Xml information specifically includes each task information, and the task information represents specifically required algorithm codes for each experiment step (for example, each step of a prediction experiment, and algorithm codes for each experiment step of a training experiment may be preset in the machine learning platform), and these algorithm codes are derived from the algorithm package. In this embodiment, the raw data on which the algorithm training depends may be referred to as source data, that is, the source data includes: the code of the algorithms in the algorithm package depends on the raw data at the time of training.
And repeating the process, sequentially executing the job tasks 2 to N, and outputting the running results of the job tasks 2 to N to a target table, wherein N is a positive integer greater than or equal to 2.
For example: and the offline computing platform submits an instruction for executing the prediction experiment to the spark in a spark-submit mode, and the spark calls the uploaded codes in the algorithm package to analyze the XML information of the experiment. When the code is operated, a job task input by an active table is found out, according to the mapping relation between the active table and a data source, a structured data file can be mapped into a database table from a corresponding HIVE (HIVE is a data warehouse tool based on Hadoop, a simple sql query function is provided, sql statements can be converted into a MapReduce task to be operated) or MQ takes out predicted data and converts the predicted data into RDD, and finally the job task is executed. And after the operation is finished, outputting the result to a target table (or called an intermediate table), and sequentially executing all job tasks downwards until the operation is finished. And at the moment, the whole prediction experiment is finished, and a final prediction result is output.
It should be noted that when executing a job task, it is necessary to distinguish whether the job task is used for model prediction or other functions. If the model file is used for model prediction, the model file needs to be deserialized, and then the prediction method of the model file is executed; otherwise, calling the corresponding algorithm. The off-line computing platform in the above steps is scheduled periodically and circularly. With the scheme as in the above example, the offline prediction scenario can be satisfied.
Wherein, the extracting source data according to the mapping relationship between the active table and the data source comprises:
and extracting the source data from the MQ according to the mapping relation of the active table and the data source. And outputting the running results of the job tasks 1 to N to the target table in an MQ mode. For example:
when the method is applied to a real-time prediction scene, the source data is loaded through the MQ, converted into the RDD and then executed with the job task, and the prediction result is output in an MQ mode. When applied to a real-time prediction scenario, the method may be specifically implemented on a system as shown in fig. 1b, and specifically, the system includes:
the system comprises a machine learning platform, a model service platform, a Nginx (or called as engine x, which is a high-performance HTTP and reverse proxy server), a cache Redis, a DB (DATABASE), a file system and a service capability node, wherein end devices of the system can establish channels through the Internet and perform data interaction through respective data transmission ports.
The DB (DATABASE) may be a server cluster for data storage and management, which is formed by multiple servers on a hardware level, specifically of a currently common DATABASE architecture type.
The service capability node comprises a plurality of computing nodes, each computing node can be a server, a computer or a virtual machine established by allocated computing resources, and the service capability node is a point for really providing services and can abstract the concept of the computing resources, such as information of CPU core number, memory capacity, disk capacity, bandwidth and the like. Meanwhile, for load and high availability, N service capability nodes are deployed simultaneously, and each node has an interface for issuing a model and a uniform interface for model service.
The cache Redis is used for caching the currently processed data so as to improve the data processing efficiency of the system.
The model service platform can be specifically a workstation, a super computer, or a server set for data processing, which is composed of a plurality of servers. The model servization platform provides an interface for publishing the model.
The file system is used for storing and managing XML information, a dependent algorithm package and a model file required by the experiment.
In this embodiment, still include:
and receiving an interface calling request sent by the machine learning platform, and calling a model interface for a model pointed by the interface calling request. And distributing service capacity nodes for the model interfaces according to the interface calling requests.
And the interface calling request is sent out after the machine learning platform is triggered to issue a model issuing process. Specifically, the machine learning platform creates a training experiment, and after the training is finished, a corresponding model file is generated and stored in a file system. Specifically, the user accesses the machine learning platform through the user terminal, or directly operates the interactive device of the machine learning platform to select the required computing resource (a CPU memory in the service capability node, etc.), and starts to publish the model. At this time, the machine learning platform calls an interface of the release model of the model service platform, where the interface call request includes: the relevant information of the model (including information of authority, module, storage path, computing resource and the like) is sent.
And then, acquiring a model file corresponding to the interface calling request from a file system, and sending the model file to the distributed service capability node. And storing the model file, the experimental data corresponding to the model file and the mapping relation of the distributed service capability nodes into a cache and a DB (DATABASE).
Specifically, after receiving the request, the model service platform selects N service capability nodes to be used for publishing the model according to a certain policy and according to the resource status of the service capability nodes. Then, the model file is downloaded from the file system and transmitted to each service capability node in a file stream mode. And after receiving the model file, the service capability node carries out deserialization on the model file and feeds back a release result. If the service capability nodes are successfully published, the model service platform stores the model, the resources and the mapping relation of the model and the resources into a cache and a DB (DATABASE).
Specifically, the user terminal triggers the prediction experiment by calling an Nginx interface. For example: the machine learning platform feeds back to the user a model predicting the high available Nginx interface address (address context unique). And calling a high-availability Nginx interface of the model prediction service by a user to initiate a prediction request. After the Nginx receives the request, the lua is used for analyzing the context of the request and caching the model information and the service capability node information which are found to be mapped. Nginx selects one of the service capability nodes according to a certain strategy and calls a prediction interface of the service capability node. And the service capability node uses a java reflection mechanism to proxy and call the prediction function of the corresponding model according to the requested model information, and returns the prediction result. It should be noted that in this process, a spark context computing environment is required. To improve the real-time performance of the prediction, the spark context uses a local mode.
For example: in practical applications, the embodiment may be specifically implemented as a process of generating and publishing a model in a real-time prediction scenario as shown in fig. 4, where the process includes:
s601: a training experiment is created. Specifically, a training experiment is established by the machine learning platform and used for cattle prevention detection, and random forest secondary classification is used for algorithm nodes.
S602: training the experiment and generating the model. Specifically, after the experiment training is finished, the random forest classification nodes generate corresponding models which are java class serialized files, a predicted function is arranged in the models, and the prediction can be achieved by calling the function.
S700: the computational resources required by the model are set and the model is published. Specifically, a user sets that the deployment model needs 10 cores of CPU and 5G of memory according to own service requirements, then selects the model, and clicks to issue.
S800: and selecting N service capability nodes as resources according to a certain strategy, downloading the model file, and sending the model file to each service capability node. Specifically, the machine learning platform calls an interface of a publishing model of the model servitization platform to publish the model. The model service platform selects N service capability nodes to deploy the model according to the load condition of the service capability node host.
S900: and after receiving the model file, deserializing the model file and feeding back a published result. Specifically, the model service platform calls the interfaces of the release models of the service capability nodes one by one to release the models. And after receiving the file stream of the model, the service capability node carries out deserialization, loads the class and finally feeds back the successful loading.
S1000: the model, the resource, and their mapping relation are stored in a cache and a DB (DATABASE). Specifically, the model service platform maps the model, N service capability nodes, modules, contexts, and other information accordingly, persists the mapped information in a DATABASE DB (DATABASE), and caches the persisted information in a cache.
S1100: the user is fed back the Nginx interface address (context unique for the address) of the model prediction service. Specifically, the machine learning platform feeds back the calling address of the model to the user, such as: http:// domain name + port/model/scalperCheck. Where, "/model/scalperCheck" is the context of the service and is unique.
Then, the user terminal triggers the prediction experiment by calling an Nginx interface, and the execution flow of the prediction experiment is as shown in fig. 5:
s1201: and calling an Nginx interface of the model prediction service and initiating a prediction request. Specifically, the user starts to call the service "http:// domain name + port/model/scalperCheck" of the model to make a prediction of whether it is a cattle or not.
S1202: and resolving the context of the request by using the lua, and finding the mapped model information and the service capability node information in the cache. Specifically, nginx will use lua to obtain the context/model/scalperCheck of the request, and find the service capability node information (e.g., 10.11.20.127, 10.11.20.128) and model information (e.g., com.su. Lag. Spark. Main. DealWithRandordForest) mapped by it in the cache.
S1203: and selecting one service capability node according to a certain strategy and calling a prediction interface of the service capability node. Specifically, nginnx selects the service node of 10.11.20.128 to provide service using a random policy. The Nginx routes the unified service interface of this service node and transparently passes the predicted data to the interface.
S1300: and according to the requested model information, calling the prediction function of the corresponding model by using a java reflection mechanism proxy, and returning the result. Specifically, after receiving the request, the service node 10.11.20.128 finds the model and calls its prediction function in reflection to predict according to two parameters, i.e., class = com.
In this embodiment, according to different real-time level requirements, an architecture scheme matched with the real-time level requirements can be adopted, so that the complexity of codes and deployment is greatly reduced, and one-key deployment and invocation of a machine learning model are realized, for example: in this embodiment, the scheme shown in fig. 3 is provided for the offline prediction and the quasi-real-time prediction scenarios, and the scheme shown in fig. 4 is provided for the real-time prediction scenarios. Therefore, the method and the device can be applied to at least three development scenes of off-line prediction, quasi-real-time prediction and real-time prediction, and the method and the device have strong universality. Compared with the traditional technical scheme that the pmml format file needs to be compiled according to a certain specification so as to be generated, the embodiment also greatly reduces the complexity of coding through a model training mode of a machine learning platform, is convenient for developers to freely customize own algorithm logic (only by providing an algorithm package and XML information of a corresponding target service system), does not need to meet the requirement of a specific specification compiling algorithm, and therefore the freedom degree of code development is improved. The scheme of the embodiment can be efficiently applied to offline, quasi-real-time and real-time scenes.
An embodiment of the present invention further provides a machine learning platform, including: the device comprises a processor, a network interface, a memory and a communication bus, wherein the communication bus is used for connection communication among all components.
The memory having stored therein a computer program for execution by the processor, the computer program executed by the processor comprising:
and the training module is used for operating a training experiment, generating a model file corresponding to the training experiment and establishing a prediction experiment according to the model file.
A preprocessing module, configured to obtain experimental data corresponding to the prediction experiment, where the experimental data includes: XML information, the algorithm package on which the prediction experiment depends.
And the processing module is used for calling the algorithm package depended by the prediction experiment to analyze the XML information, loading the analysis result to the model and then executing the prediction experiment.
In the current mainstream training and deployment scheme of machine learning models: the model is firstly saved into PMML format file, and then the PMML format file is analyzed by JPMML tool or self-defined analysis method to predict. However, the pmml mode is required, and the mode requires that the algorithm code must meet a specific specification, the rule is complex, so that the code writing is complicated, the code writing is required to be performed according to a specific format/specification, and some self-defined algorithms are difficult to write. The method is time-consuming and labor-consuming, and is difficult to further meet the real-time requirement of the current online service.
In this embodiment, according to different real-time level requirements, an architecture scheme configured with the real-time level requirements can be adopted, so that the complexity of codes and deployment is greatly reduced, and one-key deployment and calling are realized. And the complexity of coding is greatly reduced, the algorithm logic of the self is freely customized, and the requirement of meeting a specific standard writing algorithm is not required (the traditional technical scheme needs to write according to a certain standard so as to generate a pmml format file), so that the freedom of the code is realized. The whole experiment or model can complete automatic deployment and can cover various scenes such as off-line, quasi-real-time and real-time.
Furthermore, according to the mapping relation between the source table and the data source, data is taken out from the corresponding HIVE (the HIVE is slow and used for offline) or MQ (the MQ is fast and used for quasi-real-time) and converted into RDD. Specifically, according to the mapping relationship between the source table and the data source, the corresponding HIVE table or MQ is found, the data is read, and the data is converted into the RDD format. According to a specific application scene, the data source can be flexibly adjusted by switching the interface, so that the embodiment can meet different scenes such as off-line, quasi-real-time and real-time scenes.
Specifically, the processing module is specifically configured to: calling an algorithm package depended by the prediction experiment, extracting an active table from the XML information, and inputting a jobtask 1; extracting source data according to the mapping relation between the active table and the data source, converting the source data into RDD (resource description data), executing the job task 1 according to the RDD obtained by conversion, and outputting the operation result of the job task 1 to a target table; repeating the process, sequentially executing a job task 2 to a job task N, and outputting the running results of the job task 2 to the job task N to a target table, wherein N is a positive integer greater than or equal to 2.
Wherein, the XML information includes: and predicting the step flow of the experiment execution, wherein each step corresponds to one jobb task, each jobb task comprises the logic class and parameter information of the corresponding step, and the output result of the previous jobb task is used as the input of the next jobb task.
Further, the method also comprises the following steps:
the receiving module is used for receiving an interface calling request sent by the machine learning platform and calling a model interface for a model pointed by the interface calling request, wherein the interface calling request is sent out after the machine learning platform is triggered to issue a model issuing process; and distributing service capacity nodes for the model interfaces according to the interface calling requests.
The distribution module is used for acquiring the model file corresponding to the interface calling request from a file system and sending the model file to the distributed service capability node; and storing the model file, the experimental data corresponding to the model file and the mapping relation of the distributed service capacity nodes into a cache and a database.
The embodiment is applied to two scenes of off-line prediction and quasi-real-time prediction, has the capability of processing large data volume, and is very suitable for scenes with large user volume; the real-time prediction scheme can realize the uniform allocation and scheduling of resources according to the application scene of a user so as to facilitate parallel expansion, and is particularly suitable for scenes with seasonal, holiday and sudden access surge.
All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to some descriptions of the method embodiment for relevant points. The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are also within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A method of processing a machine learning model, comprising:
running a training experiment through a machine learning platform, generating a model file corresponding to the training experiment, and creating a prediction experiment according to the model file, wherein the model file is deserialized to obtain a model, and a parent class of the model is provided with an interface corresponding to the prediction experiment;
acquiring, by the machine learning platform, experimental data corresponding to the prediction experiment, the experimental data including: XML information, the algorithm package on which the prediction experiment depends;
calling an algorithm package depended by the prediction experiment to analyze the XML information, and executing the prediction experiment after an analysis result is loaded to the model;
the XML information comprises: the step flow of the execution of the prediction experiment is carried out, wherein each step corresponds to one jobtask, each jobtask comprises the logic class and parameter information of the corresponding step, and the output result of the previous jobtask is used as the input of the next jobtask;
after the analysis result is loaded to the model, the prediction experiment is executed, including: calling an algorithm package depended by the prediction experiment, extracting an active table from the XML information, and inputting a jobtask 1; extracting source data according to the mapping relation between the active table and the data source, converting the source data into an elastic distributed data set RDD, executing the job task 1 according to the RDD obtained by conversion, and outputting the operation result of the job task 1 to a target table; and repeating the process, sequentially executing the job tasks 2 to N, and outputting the running results of the job tasks 2 to N to a target table, wherein N is a positive integer greater than or equal to 2.
2. The method of claim 1, further comprising: and when the model passes the prediction experiment, releasing the model, wherein the model is used for establishing the robot in a target business system.
3. The method of claim 1, wherein extracting source data according to the mapping relationship between the active table and the data source comprises:
extracting the source data from a message queue MQ according to the mapping relation of the active table and a data source;
and outputting the running results of the job tasks 1 to N to the target table in an MQ mode.
4. The method of claim 1, further comprising:
receiving an interface calling request sent by the machine learning platform, and calling a model interface for a model to which the interface calling request points, wherein the interface calling request is sent out after the machine learning platform is triggered to issue a model issuing process;
and distributing service capacity nodes for the model interfaces according to the interface calling requests.
5. The method of claim 4, further comprising:
obtaining a model file corresponding to the interface calling request from a file system, and sending the model file to the distributed service capability node;
and storing the model file, the experimental data corresponding to the model file and the mapping relation of the distributed service capacity nodes into a cache and a database.
6. The method of claim 1, further comprising:
and the user terminal triggers the prediction experiment by calling an Nginx interface.
7. An apparatus for processing a machine learning model, comprising:
the system comprises a processor, a network interface, a memory and a communication bus, wherein the communication bus is used for connection and communication among all components;
the memory having stored therein a computer program for execution by the processor, the computer program executed by the processor comprising:
the training module is used for operating a training experiment, generating a model file corresponding to the training experiment and establishing a prediction experiment according to the model file;
a preprocessing module, configured to obtain experimental data corresponding to the prediction experiment, where the experimental data includes: XML information, an algorithm package on which the prediction experiment depends;
the processing module is used for calling the algorithm package depended by the prediction experiment to analyze the XML information, and executing the prediction experiment after an analysis result is loaded to the model;
the XML information comprises: the step flow of the execution of the prediction experiment is carried out, wherein each step corresponds to one jobtask, each jobtask comprises the logic class and parameter information of the corresponding step, and the output result of the previous jobtask is used as the input of the next jobtask;
after the analysis result is loaded to the model, the prediction experiment is executed, including: calling an algorithm package depended by the prediction experiment, extracting an active table from the XML information, and inputting a jobtask 1; extracting source data according to the mapping relation between the active table and the data source, converting the source data into an elastic distributed data set RDD, executing the job task 1 according to the RDD obtained through conversion, and outputting the operation result of the job task 1 to a target table; and repeating the process, sequentially executing the job tasks 2 to N, and outputting the running results of the job tasks 2 to N to a target table, wherein N is a positive integer greater than or equal to 2.
8. The apparatus of claim 7,
further comprising:
the receiving module is used for receiving an interface calling request sent by a machine learning platform and calling a model interface for a model to which the interface calling request points, wherein the interface calling request is sent out after the machine learning platform is triggered to issue a model issuing process; distributing service capacity nodes to the model interfaces according to the interface calling requests;
the distribution module is used for acquiring the model file corresponding to the interface calling request from a file system and sending the model file to the distributed service capability node; and storing the model file, the experimental data corresponding to the model file and the mapping relation of the distributed service capacity nodes into a cache and a database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810258874.5A CN108510082B (en) | 2018-03-27 | 2018-03-27 | Method and device for processing machine learning model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810258874.5A CN108510082B (en) | 2018-03-27 | 2018-03-27 | Method and device for processing machine learning model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108510082A CN108510082A (en) | 2018-09-07 |
| CN108510082B true CN108510082B (en) | 2022-11-11 |
Family
ID=63378585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810258874.5A Active CN108510082B (en) | 2018-03-27 | 2018-03-27 | Method and device for processing machine learning model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108510082B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299785B (en) * | 2018-09-17 | 2022-04-26 | 浪潮软件股份有限公司 | Method and device for realizing machine learning model |
| CN109407997B (en) * | 2018-11-09 | 2021-04-23 | 长沙理工大学 | Data processing method, device and equipment and readable storage medium |
| CN111488326B (en) * | 2019-01-25 | 2023-04-07 | 阿里巴巴集团控股有限公司 | MapReduce job execution method, node device and storage medium |
| CN109872819A (en) * | 2019-01-30 | 2019-06-11 | 杭州脉兴医疗科技有限公司 | A kind of acute kidney injury incidence rate forecasting system based on Intensive Care Therapy detection |
| CN110308910B (en) * | 2019-05-30 | 2023-10-31 | 苏宁金融服务(上海)有限公司 | Method, device and computer equipment for deploying algorithm model and monitoring risk |
| CN110555550B (en) * | 2019-08-22 | 2023-06-23 | 创新先进技术有限公司 | Online prediction service deployment method, device and equipment |
| CN110490334A (en) * | 2019-08-27 | 2019-11-22 | 上海丙晟科技有限公司 | A kind of machine learning of low latency is the generation method of service |
| CN110704135B (en) * | 2019-09-26 | 2020-12-08 | 北京智能工场科技有限公司 | Competition data processing system and method based on virtual environment |
| EP4027584A4 (en) * | 2019-09-30 | 2022-08-31 | Huawei Technologies Co., Ltd. | Device information determination method and apparatus, and system |
| CN110837896B (en) * | 2019-11-22 | 2022-07-08 | 中国联合网络通信集团有限公司 | Storage and calling method and device of machine learning model |
| CN110852449B (en) * | 2019-11-25 | 2023-11-14 | 北京百度网讯科技有限公司 | Model migration method and electronic equipment |
| CN111062521B (en) * | 2019-11-29 | 2023-09-12 | 微民保险代理有限公司 | Online prediction method, system and server |
| CN111273901B (en) * | 2020-01-20 | 2022-09-09 | 浙江邦盛科技股份有限公司 | File format and deployment method of machine learning model capable of being rapidly deployed online |
| CN111523676B (en) * | 2020-04-17 | 2024-04-12 | 第四范式（北京）技术有限公司 | Method and device for assisting machine learning model to be online |
| US11775757B2 (en) | 2020-05-04 | 2023-10-03 | International Business Machines Corporation | Automated machine-learning dataset preparation |
| CN112464569A (en) * | 2020-12-10 | 2021-03-09 | 北京明略软件系统有限公司 | Machine learning method and system |
| CN113515895B (en) * | 2021-07-30 | 2024-03-01 | 北京中网易企秀科技有限公司 | Cross-platform model prediction method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104933075A (en) * | 2014-03-20 | 2015-09-23 | 百度在线网络技术（北京）有限公司 | User attribute predicting platform and method |
| CN105005575A (en) * | 2015-03-05 | 2015-10-28 | 张良均 | Quick developing interface method for enterprise intelligent prediction |
| CN106250987A (en) * | 2016-07-22 | 2016-12-21 | 无锡华云数据技术服务有限公司 | A kind of machine learning method, device and big data platform |
| CN106776326A (en) * | 2016-12-20 | 2017-05-31 | 中国农业银行股份有限公司 | The modeling method and system of a kind of Data Analysis Model |
-
2018
- 2018-03-27 CN CN201810258874.5A patent/CN108510082B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104933075A (en) * | 2014-03-20 | 2015-09-23 | 百度在线网络技术（北京）有限公司 | User attribute predicting platform and method |
| CN105005575A (en) * | 2015-03-05 | 2015-10-28 | 张良均 | Quick developing interface method for enterprise intelligent prediction |
| CN106250987A (en) * | 2016-07-22 | 2016-12-21 | 无锡华云数据技术服务有限公司 | A kind of machine learning method, device and big data platform |
| CN106776326A (en) * | 2016-12-20 | 2017-05-31 | 中国农业银行股份有限公司 | The modeling method and system of a kind of Data Analysis Model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108510082A (en) | 2018-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108510082B (en) | Method and device for processing machine learning model | |
| CN107590001B (en) | Load balancing method and device, storage medium and electronic equipment | |
| CN109074377B (en) | Managed function execution for real-time processing of data streams | |
| EP3667500B1 (en) | Using a container orchestration service for dynamic routing | |
| EP3404542A1 (en) | Data pipeline architecture for analytics processing stack | |
| JP2022013649A (en) | Streaming computing method and device based on dag interaction | |
| CN110716744A (en) | Data stream processing method, system and computer readable storage medium | |
| EP4264427A1 (en) | Multi-tenant control plane management on computing platform | |
| US11321090B2 (en) | Serializing and/or deserializing programs with serializable state | |
| CN112306586A (en) | Data processing method, device, equipment and computer storage medium | |
| CN109614227A (en) | Task resource concocting method, device, electronic equipment and computer-readable medium | |
| CN110764752A (en) | System and method for realizing graphical service arrangement of Restful service based on micro-service architecture | |
| CN112052011A (en) | Method and device for combining small programs, electronic equipment and medium | |
| CN111309341B (en) | Android application installation flow optimization method based on time-consuming prediction | |
| CN116775041A (en) | Big data real-time decision engine based on stream computing framework and RETE algorithm | |
| CN116668520A (en) | Gateway-based service arrangement method, system, equipment and storage medium | |
| CN110764769A (en) | Method and device for processing user request | |
| CN115794262A (en) | Task processing method, device, equipment, storage medium and program product | |
| CN113779122B (en) | Method and device for exporting data | |
| US11556382B1 (en) | Hardware accelerated compute kernels for heterogeneous compute environments | |
| CN112637288A (en) | Streaming data distribution method and system | |
| US20200293386A1 (en) | Messaging abstraction layer for integration with message oriented middleware platforms | |
| CN116541133B (en) | Nanotube method for container application, device and electronic equipment thereof | |
| CN113556375A (en) | Cloud computing service method and device, electronic equipment and computer storage medium | |
| CN112559001A (en) | Method and device for updating application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |