CN114091687A

CN114091687A - Method and device for automatic delivery and deployment of machine learning model

Info

Publication number: CN114091687A
Application number: CN202111406169.3A
Authority: CN
Inventors: 童楚婕
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2021-11-24
Filing date: 2021-11-24
Publication date: 2022-02-25

Abstract

The invention discloses a method and a device for automatic delivery and deployment of a machine learning model, which relate to the technical field of artificial intelligence, and the method comprises the following steps: developing a source code for constructing a machine learning pipeline according to the characteristic data of the machine learning model to be constructed; creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model; and deploying the target machine learning model to a production environment for operation and performance monitoring. The invention can realize the rapid delivery and deployment of the machine learning model, improves the delivery and deployment efficiency of the machine learning model and can provide better service for users.

Description

Method and device for automatic delivery and deployment of machine learning model

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to a method and a device for automatic delivery and deployment of a machine learning model.

Background

This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

At present, with the development of new technologies such as artificial intelligence, data science and machine learning are becoming core abilities to solve complex realistic problems, to transform industries, and to create value in all fields. Many enterprises establish internal data science teams, build machine learning capacity and develop prediction models capable of bringing business values to users. Data scientists can build predictive ML (Machine Learning) models off-line on training data sets, but the real challenge is not to build Machine Learning models, but rather to build integrated Machine Learning systems to automatically generate Machine Learning models and keep the system running continuously in production.

At present, the delivery and deployment of the machine learning model are mostly realized in a manual mode. As shown in fig. 1, fig. 1 illustrates the most common machine learning data scientist today, manually performing the process of building a machine learning model and using the model as a predictive service: this manually driven process by the data scientist may be sufficient when the model is rarely changed or trained. However, because the process has many features, such as each step involved in the script-driven and interactive processes is manual, the process of establishing the model in the conventional manual manner is too complicated, the rapid delivery and deployment of the machine learning model are difficult to achieve, and the delivery and deployment efficiency of the machine learning model is seriously reduced.

Generally, the current software system is generally realized by the flow of DevOps. However, the online operation of the machine learning model system is different from other software systems, for example, in the development aspect, the machine learning model is experimental in nature, and different functions, algorithms, modeling techniques and parameter configurations are often required to be tried based on online data so as to find the most suitable method for solving the problem as soon as possible. However, the traditional DevOps method cannot be changed greatly frequently once the system is on-line; on the other hand, on the test, in addition to the typical unit test and the integrated test, the machine learning model also requires data verification, trained model quality assessment and model verification, which cannot be done by the conventional DevOps method. Therefore, the conventional DevOps process cannot meet the requirements of what and deployment the machine learning model delivers.

Disclosure of Invention

The embodiment of the invention provides a method for automatically delivering and deploying a machine learning model, which is used for realizing the rapid delivery and deployment of the machine learning model, improving the delivery and deployment efficiency of the machine learning model and providing better service for a user, and comprises the following steps:

developing a source code for constructing a machine learning pipeline according to the characteristic data of the machine learning model to be constructed; the machine learning pipeline is used for realizing the processes of constructing and training the machine learning model;

creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration task comprises: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool;

constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model;

deploying the target machine learning model to a production environment for operation and performance monitoring;

and when abnormal data occur in performance monitoring, the steps are repeatedly executed, the updated target machine learning model is obtained according to the obtained updated data of the characteristic data of the machine learning model to be constructed, and the abnormal data do not occur in the performance monitoring of the updated target machine learning model.

The embodiment of the invention also provides a device for automatically delivering and deploying the machine learning model, which is used for realizing the rapid delivery and deployment of the machine learning model, improving the delivery and deployment efficiency of the machine learning model and providing better service for users, and the device comprises:

the source code development module is used for developing and constructing a source code of the machine learning pipeline according to the characteristic data of the machine learning model to be constructed; the machine learning pipeline is used for realizing the processes of constructing and training the machine learning model;

the machine learning pipeline building module is used for creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration task comprises the following steps: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool;

the target machine learning model building module is used for deploying the target machine learning model to a production environment and carrying out performance test;

the performance monitoring module is used for deploying the target machine learning model to a production environment for operation and performing performance monitoring;

The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the computer program, the method for automatically delivering and deploying the machine learning model is realized.

The embodiment of the invention also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the automatic delivery and deployment method of the machine learning model.

An embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, and when executed by a processor, the computer program implements the method for automatically delivering and deploying the machine learning model.

In the embodiment of the invention, according to the characteristic data of the machine learning model to be constructed, the source code of the machine learning pipeline is developed and constructed; the machine learning pipeline is used for realizing the processes of constructing and training the machine learning model; creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration task comprises: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model; deploying the target machine learning model to a production environment for operation and performance monitoring; when abnormal data occur in performance monitoring, the steps are repeatedly executed, the updated target machine learning model is obtained according to the obtained updated data of the characteristic data of the machine learning model to be built, and the abnormal data do not occur until the performance monitoring of the updated target machine learning model; meanwhile, by using the continuous integration tool, the continuous delivery tool and the continuous deployment tool, the automatic delivery, deployment and performance test of the machine learning model are realized, the problem that the traditional DevOps flow cannot be applied to the machine learning model establishment in the prior art is solved, and better service can be provided for users.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:

FIG. 1 is a schematic flow chart illustrating a method for manually delivering and deploying a machine learning model according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating an exemplary method for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating an exemplary implementation of a method for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating an exemplary method for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an apparatus for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating an exemplary implementation of an apparatus for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating an exemplary implementation of an apparatus for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 8 is a diagram illustrating an exemplary implementation of an apparatus for automatic delivery and deployment of a machine learning model according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating an exemplary implementation of an apparatus for automatic delivery and deployment of a machine learning model according to an embodiment of the present disclosure;

FIG. 10 is a diagram of a computer device for automatic delivery and deployment of machine learning models, in an embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

The term "and/or" herein merely describes an associative relationship, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are used in an open-ended fashion, i.e., to mean including, but not limited to. Reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the various embodiments is provided to illustrate the practice of the present application, and the sequence of steps is not limited thereto and can be adjusted as needed.

Embodiments of the present invention relate to the following terms, which are explained as follows:

DevOps (a combination of Development and Operations) is a collective term for a set of processes, methods and systems for facilitating communication, collaboration and integration between Development (application/software engineering), technical Operations and Quality Assurance (QA) departments. IT is a culture, exercise or practice that attaches importance to the communication and cooperation between "software developers (Dev)" and "IT operation and maintenance technicians (Ops)". Through the automatic software delivery and architecture change processes, the software can be built, tested and released more quickly, frequently and reliably.

Persistent integration CI (continuous integration): the process of analysis, construction, testing and deployment continues. The CI checks the quality of the code and tests the business logic of the product before release to the production system. It is emphasized that the build, (unit) test is performed immediately after the developer has submitted the new code. And determining whether the new code and the original code can be correctly integrated according to the test result.

Continuous delivery of CD (continuous delivery) is a software engineering technique, which allows the production process of software products to be completed within a short period, so as to ensure that the software can be stably and continuously released at any time. It aims to make the construction, testing and release of software faster and more frequent. This way, the cost and time of software development can be reduced, and the risk is reduced.

Continuous deployment of cds (continuous deployment): continuous deployment emphasizes the integration of new software functionality through automated deployment means. The method is oriented to stable function updating after online release. Automatic deployment without manual intervention can ensure that the newly added function is released to the production environment at the first time and is used by the user as soon as possible

At present, with the development of new technologies such as artificial intelligence, data science and machine learning are becoming core abilities to solve complex realistic problems, to transform industries, and to create value in all fields. Many enterprises establish internal data science teams, build machine learning capacity and develop prediction models capable of bringing business values to users. Data scientists can build ML models with predictive capabilities on training data sets offline, but the real challenge is not to build ML models, but to build integrated ML systems and keep them running continuously in production.

The online operation of the machine learning system is different from other software systems, for example, in the development aspect, a machine learning model is experimental in nature, and different functions, algorithms, modeling techniques and parameter configurations are often tried on the basis of online data so as to find the most suitable method for solving the problem as soon as possible, but once the traditional software system is online, the traditional software system cannot be changed greatly frequently; on the other hand, on the test, in addition to the typical unit test and the integrated test, the machine learning model requires data verification, trained model quality evaluation, and model verification.

Thus, there are significant differences between the automation of the machine learning model building process and the traditional DevOps process, such as persistent integration CI no longer being only about test and verification code and components, but also test and verification data, data patterns and models; continuous delivery is no longer a single software package or service, but a system (ML training model pipe) that should automatically deploy another service (model prediction service); continuous training CT is a new attribute unique to machine learning systems, involving re-automated training and service models. The traditional DevOps process cannot meet the requirements of the machine learning model, and a new construction process is needed to meet the new requirements.

Fig. 1 illustrates the process of machine learning data scientists, most commonly today, manually performing the ML step and using the model as a prediction service, which may be sufficient when the model is rarely changed or trained. The process has many characteristics, such as that each step involved in the script-driven and interactive processes is manual, the online release version of the model is not frequent, no configuration item is provided, and no CD exists, and the deployment process only refers to that the trained model is deployed as a prediction service (for example, a micro-service with REST API), and the whole ML system is not involved; and lack of effective performance monitoring.

In order to solve the above problem, an embodiment of the present invention provides a method for automatically delivering and deploying a machine learning model, so as to implement quick delivery and deployment of the machine learning model, improve delivery and deployment efficiency of the machine learning model, and provide better service for a user, where, referring to fig. 2, the method may include:

step 201: developing a source code for constructing a machine learning pipeline according to the characteristic data of the machine learning model to be constructed; the machine learning pipeline is used for realizing the process of constructing and training the machine learning model;

step 202: creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration tasks include: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool;

step 203: constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model;

step 204: deploying the target machine learning model to a production environment for operation and performance monitoring;

In the embodiment of the invention, a source code for constructing a machine learning pipeline is developed according to the characteristic data of a machine learning model to be constructed; the machine learning pipeline is used for realizing the process of constructing and training the machine learning model; creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration tasks include: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model; deploying the target machine learning model to a production environment for operation and performance monitoring; when abnormal data occur in performance monitoring, the steps are repeatedly executed, the updated target machine learning model is obtained according to the obtained updated data of the characteristic data of the machine learning model to be built, and the abnormal data do not occur until the performance monitoring of the updated target machine learning model; meanwhile, by using the continuous integration tool, the continuous delivery tool and the continuous deployment tool, the automatic delivery, deployment and performance test of the machine learning model are realized, the problem that the traditional DevOps flow cannot be applied to the machine learning model establishment in the prior art is solved, and better service can be provided for users.

In specific implementation, firstly, according to the characteristic data of a machine learning model to be constructed, a source code for constructing a machine learning pipeline is developed; the machine learning pipeline is used for realizing the process of constructing and training the machine learning model.

In an embodiment, the method may further include: acquiring feature data of a machine learning model to be constructed from a preset feature data database; the characteristic data database stores characteristic data corresponding to different machine learning models to be built in advance.

In the above embodiment, the feature data database is a centralized repository in which the definition, storage and access of normalized feature data can be performed to more quickly perform the training of the machine learning model.

In one embodiment, before the machine learning pipeline starts a new round of experimental iteration, a data scientist may manually perform data analysis on feature data, the input of this step is data to be trained and features of a feature library, and available data subjected to analysis judgment is output, specifically, feature data of a machine learning model to be built can be obtained from a preset feature data database.

In the above embodiment, the source code of the built machine learning pipeline is developed, and after the model is analyzed manually by a data scientist, the source code is pushed to a source code warehouse, and the source code for building different machine learning pipelines is stored in the source code warehouse. In this step, the relevant staff can design the experimental steps and try repeatedly the new ML algorithm of the machine learning model to be built, which helps to achieve fast iteration of the model. Meanwhile, the experimental operation in the step has symmetry, and the model of the pipeline implementation used in the development or experimental environment can be used in the pre-production and production environment, which is a key aspect of the practice of MLOps (Machine Learning Operations) unified DevOps. The phase output is the source code of the ML pipeline step, which is then pushed to the source code repository.

During specific implementation, after a source code for constructing a machine learning pipeline is developed according to characteristic data of a machine learning model to be constructed, a continuous integration task is created according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration tasks include: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; and continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool.

In an embodiment, executing the persistent integration task, performing the deployment pipeline process, and constructing the machine learning pipeline may include, as shown in fig. 3:

step 301: executing a continuous integration task, performing deployment pipeline processing, and constructing a pipeline component of the machine learning pipeline; the pipeline assembly comprises a pipeline program package, a pipeline executable file and a pipeline workpiece;

step 302: and constructing and training a machine learning model to be constructed according to the characteristic data through a pipeline component of the machine learning pipeline deployed to the target environment to obtain the target machine learning model.

In one embodiment, the method may further include: testing and processing the constructed machine learning pipeline through a continuous integration tool; the test process comprises: testing whether machine learning model training performed by the machine learning pipeline converges, whether the implementation effect of each pipeline component of the machine learning pipeline meets the preset requirement, and whether the integration degree among the testing pipeline components exceeds a preset value;

deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline may include:

and when the test processing is passed, deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline.

In one embodiment, source code submitted by a user may first be pushed to a source code repository while the pipeline components of the machine learning pipeline will be built, tested, and packaged. In addition to building the above pipeline packages and pipeline executables, the engineering logic for building a machine-learned pipeline can be tested according to the CI flow of the conventional DevOps flow for unit testing, testing whether model training has converged, testing each component in the pipeline to produce the desired workpiece, testing the integration between pipeline components.

In the above embodiments, the conduit components generated by the persistent integration tool may be deployed to the target environment. The ML pipeline in production continues to provide new pipeline implementations to the target environment, providing predictive services for new models trained with new data, the output of this stage being deployed pipeline using the new implementations of the models.

In a specific embodiment, before deploying the target machine learning model to the production environment, the method further includes: and carrying out model registration on the target machine learning model.

In an embodiment, the deployed machine learning pipeline may be automatically executed in production as triggered by a preset schedule or actual requirements. The output of this stage is the trained model, and the model is pushed to the model registry for model registration.

In specific implementation, after the machine learning pipeline is deployed to a target environment corresponding to the machine learning pipeline, the machine learning pipeline deployed to the target environment is used for constructing and training a machine learning model to be constructed according to the characteristic data to obtain the target machine learning model.

In an embodiment, the model trained by the machine learning pipeline can be used as a prediction service for prediction of a machine learning algorithm. The output of this phase is the prediction service implemented by the deployed model.

In specific implementation, the machine learning model to be constructed is constructed and trained through a machine learning pipeline deployed to a target environment according to the characteristic data to obtain a target machine learning model, and then the target machine learning model is deployed to a production environment to operate and monitor performance; and when abnormal data occur in performance monitoring, the steps are repeatedly executed, the updated target machine learning model is obtained according to the obtained updated data of the characteristic data of the machine learning model to be constructed, and the abnormal data do not occur in the performance monitoring of the updated target machine learning model.

In an embodiment, deploying the target machine learning model to the production environment for running operations may include: deploying the target machine learning model to a production environment for monitoring the prediction service; when abnormal data occurs in performance monitoring, the above steps are repeatedly executed, an updated target machine learning model is obtained according to the obtained updated data of the feature data of the machine learning model to be constructed, and until the abnormal data does not occur in the performance monitoring of the updated target machine learning model, the method may include: when the monitoring of the prediction service is wrong, for example, the model cannot achieve the expected function, the actual machine learning performance of the model cannot be matched with the expected value, and the like.

In one embodiment, after training to obtain a machine learning model, the model is automatically subjected to data validation and model validation steps to evaluate the prediction quality of the model, and an evaluation metric value generated by the newly trained model is compared with the current model to ensure that the performance of the model is consistent across various parts of the data, ensure the infrastructure compatibility and consistency of the new model with the prediction service API.

In the above embodiment, when abnormal data occurs in performance monitoring, the above steps are repeatedly executed, and the updated target machine learning model is obtained according to the acquired updated data of the feature data of the machine learning model to be constructed until no abnormal data occurs in performance monitoring of the updated target machine learning model, which can be realized by a continuous integration tool, a continuous delivery tool, and a continuous deployment tool, so that the DevOps principle can be applied to the method of the ML system (MLOps). The novel machine learning model can be retrained and deployed automatically by automatically training and deploying the ML pipeline, the CI/CD system is arranged to realize automatic testing and deploying the novel pipeline, automation and monitoring in all steps of ML system construction are realized, the steps comprise integration, testing, publishing, deploying and infrastructure management, the training, verification, testing, publishing and deploying processes of the whole machine learning model can be automated, quick delivery of the ML system can be realized, and the efficiency is improved.

In the above embodiment, after the new machine learning model is trained on the new feature data, the steps of automatic data verification and machine learning model verification are performed to ensure the verification operation on the new feature data, so as to evaluate the prediction quality of the machine learning model, and the evaluation metric value generated by the newly trained machine learning model is compared with the current model to ensure that the performance of the model is consistent on each part of the data, and ensure the infrastructure compatibility and the consistency with the prediction service API of the new model.

In the embodiment, by using the continuous integration tool, the continuous delivery tool and the continuous deployment tool, the automatic delivery, deployment and performance test of the machine learning model are realized, the problem that the traditional DevOps flow cannot be applied to the machine learning model establishment in the prior art is solved, and better service can be provided for users. Technical personnel need to integrate the training and deployment processes of the existing machine learning model, combine with the method in the embodiment of the invention, develop the whole set of automatic processes in the embodiment of the invention, and connect the whole set of automatic processes in series, so as to realize automatic continuous training, deployment, continuous delivery and continuous integration online.

In specific implementation, the method for automatically delivering and deploying a machine learning model provided by the embodiment of the present invention, as shown in fig. 4, may further include:

step 401: adding version identification for each constructed machine learning pipeline;

step 402: obtaining each pipe component in the machine learning pipe: the time spent in the process of constructing and training the machine learning model to be constructed;

step 403: and recording the version identification and the duration aiming at the machine learning pipeline constructed each time.

In an embodiment, reproducibility and comparison is achieved by recording information about each execution of the ML pipeline. Debugging errors and exceptions may also be facilitated. Each time a pipeline is executed, the ML metadata store records the start and end dates, time, and time taken by the pipeline to complete each step, the version of the pipeline and components executed, and the model evaluation metrics generated for the training set and the test set in the model evaluation step.

In specific implementation, the method for automatically delivering and deploying a machine learning model provided by the embodiment of the present invention may further include: monitoring a process of executing a prediction service on a target machine learning model deployed to a production environment;

and when the preset abnormal service condition occurs in the process, re-executing the steps of creating and executing the continuous integration task through the continuous integration tool until the abnormal service condition does not occur in the process of executing the prediction service by the target machine learning model.

In an embodiment, statistical information about model performance is collected from real-time data, and the output of this stage is the trigger conditions for executing the pipeline or executing a new machine learning experiment cycle, and then based on these trigger conditions, the operation of generating the machine learning pipeline or generating the machine learning model is triggered.

A specific example is given below to illustrate a specific application of the method of the present invention, which may include the following steps:

1. before a pipeline starts a new round of experiment iteration, a data scientist manually analyzes feature data of the data, the input of the step is the feature data needing training, and the available feature data after analysis and judgment are output;

2. automatic experiment: the step of receiving the analyzed available feature data, outputting source codes of the ML pipeline, and pushing the source codes to a source code warehouse after the analysis of a machine learning model manually operated by a data scientist;

3. continuous integration: receiving the source code in the step 2, constructing a machine learning channel corresponding to the source code and running various tests, wherein the output of the stage is pipeline components (such as a pipeline program package, a pipeline executable file and a pipeline workpiece) to be deployed at a later stage;

4. pipeline continuous delivery: deploying the artifact generated by the continuous integration phase to the target environment, the output of this phase being a deployed machine learning pipeline using the new implementation of the model;

5. pipeline automation: and triggering to automatically execute the machine learning pipeline in the production environment according to the response condition of the trigger. The characteristics and trigger conditions of the characteristic warehouse and the machine learning pipeline deployed in the step 3 are input in the step, and the previous execution information (used for model evaluation) of the machine learning model is combined, the output of the step is the trained machine learning model, the machine learning model is pushed to a model registry, and meanwhile the execution information of the machine learning pipeline is output to a metadata management module, and the metadata management module is used for recording the information about each execution of the ML pipeline to realize reproducibility and comparison. Debugging errors and exceptions may also be facilitated. Each time a pipeline is executed, the ML metadata store records start and end dates, time spent by the pipeline in each step, versions of the executed pipeline and components, model evaluation indexes generated for a training set and a test set in the model evaluation step, and other metadata;

6. model continuous delivery: and taking the machine learning model trained in the step 5 as a prediction service, wherein the output of the stage is the prediction service of the deployed machine learning model.

7. Monitoring: the prediction service in 6 is subjected to real-time data collection, and information about the performance of the machine learning model is counted, and the output of the stage is a trigger for performing reconstruction of the machine learning pipeline and/or performing a new machine learning experiment cycle.

As described above, this example provides a method of applying the DevOps principle to ML systems (MLOps). The method can automatically retrain and deploy the new machine learning model by automatically training and deploying the ML pipeline, simultaneously set the CI/CD system to realize automatic testing and deploying the new pipeline, realize automation and monitoring in all steps of ML system construction, realize quick iteration and delivery of the ML model and the ML system, improve efficiency and deal with quick change in data and service environment.

Of course, it is understood that other variations of the above detailed flow can be made, and all such variations are intended to fall within the scope of the present invention.

In the embodiment of the invention, a source code for constructing a machine learning pipeline is developed according to the characteristic data of a machine learning model to be constructed; the machine learning pipeline is used for realizing the process of constructing and training the machine learning model; according to the source code, carrying out deployment pipeline processing and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model; deploying the target machine learning model to a production environment to test the performance of the service; when the performance test is failed, creating and executing a continuous integration task through a continuous integration tool until the target machine learning model passes the performance test; the continuous integration tasks include: continuously executing the steps of constructing the machine learning pipeline and constructing and training a machine learning model to be constructed through a continuous delivery tool to continuously obtain a target machine learning model; compared with the technical scheme of manually constructing the machine learning model in the prior art, the method has the advantages that the constructed machine learning pipeline can automatically realize construction and training of the machine learning model, the problem that the traditional manual modeling process is too complicated in the prior art is solved, the machine learning model can be rapidly delivered and deployed, and the delivery and deployment efficiency of the machine learning model is improved; meanwhile, by using the continuous integration tool, the continuous delivery tool and the continuous deployment tool, the automatic delivery, deployment and performance test of the machine learning model are realized, the problem that the traditional DevOps flow cannot be applied to the machine learning model establishment in the prior art is solved, and better service can be provided for users.

As described above, embodiments of the present invention propose a method of applying the DevOps principle to ML systems (MLOps). The new model of machine learning can be retrained and deployed automatically by automatically training and deploying the ML pipeline, meanwhile, the CI/CD system is set to realize automatic testing and deploying the new pipeline, and automation and monitoring in all steps of ML system construction are realized, and the method can comprise integration, testing, release, deployment and infrastructure management.

The embodiment of the invention has the following advantages:

1. the method provided by the invention automates the training, verifying, testing, releasing and deploying processes of the whole machine learning model, can realize the quick delivery of the ML system, and improves the efficiency;

2. iteration of a new model and service can be quickly realized, the new model is continuously optimized, rapid changes in data and service environments are responded, and better service is provided for users;

3. more users and scenes are served, and the generalization of a machine learning model is realized.

The embodiment of the invention also provides a device for automatic delivery and deployment of the machine learning model, which is described in the following embodiment. Because the principle of the device for solving the problems is similar to the method for automatically delivering and deploying the machine learning model, the implementation of the device can be referred to the implementation of the method for automatically delivering and deploying the machine learning model, and repeated details are omitted.

An embodiment of the present invention further provides an apparatus for automatically delivering and deploying a machine learning model, so as to implement rapid delivery and deployment of the machine learning model, improve delivery and deployment efficiency of the machine learning model, and provide a better service for a user, as shown in fig. 5, the apparatus includes:

the source code development module 501 is configured to develop a source code for constructing a machine learning pipeline according to feature data of a machine learning model to be constructed; the machine learning pipeline is used for realizing the process of constructing and training the machine learning model;

a machine learning pipeline building module 502, configured to create a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration tasks include: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool;

a target machine learning model building module 503, configured to build and train a machine learning model to be built according to the feature data through a machine learning pipeline deployed in a target environment, so as to obtain a target machine learning model;

a performance monitoring module 504, configured to deploy the target machine learning model to a production environment for operation and perform performance monitoring;

In one embodiment, the machine learning conduit build module is specifically configured to:

executing a continuous integration task, performing deployment pipeline processing, and constructing a pipeline component of the machine learning pipeline; the pipeline component comprises a pipeline program package, a pipeline executable file and a pipeline workpiece;

and constructing and training a machine learning model to be constructed according to the characteristic data through a pipeline component of the machine learning pipeline deployed to the target environment to obtain the target machine learning model.

In one embodiment, as shown in fig. 6, further includes:

a test processing module 505 for

Testing and processing the constructed machine learning pipeline through a continuous integration tool; the test process comprises: testing whether machine learning model training performed by the machine learning pipeline converges, whether the implementation effect of each pipeline component of the machine learning pipeline meets the preset requirement, and whether the integration degree among the testing pipeline components exceeds a preset value;

the machine learning pipeline building module is specifically used for:

In one embodiment, as shown in fig. 7, further includes:

a model registration module 506 to:

and carrying out model registration on the target machine learning model.

In one embodiment, as shown in fig. 8, further includes:

a characteristic data obtaining module 507, configured to:

acquiring feature data of a machine learning model to be constructed from a preset feature data database; the characteristic data database is prestored with characteristic data corresponding to different machine learning models to be constructed.

In one embodiment, further comprising:

a recording module to:

adding version identification for each constructed machine learning pipeline;

obtaining each pipe component in the machine learning pipe: the time spent in the process of constructing and training the machine learning model to be constructed;

and recording the version identification and the duration aiming at the machine learning pipeline constructed each time.

In one embodiment, further comprising:

a service anomaly monitoring module for:

monitoring a process of executing a prediction service on a target machine learning model deployed to a production environment;

A specific embodiment is given below to illustrate a specific application of the apparatus of the present invention, and as shown in fig. 9, the embodiment may include the following:

1. designing an experiment module: the module can realize the quick iteration of the machine learning experiment in the design experiment step and repeatedly try a new ML algorithm and a new model. While the experimental operation is symmetrical, the pipeline implemented machine learning pipeline used in the development or experimental environment can be used in the pre-production and production environments, which is a key aspect of the practice of MLOps unified DevOps. The output of this stage is the source code of the ML pipeline step, which is then pushed to the source code repository;

2. continuous integration module of pipeline: this module user submits or pushes new code to the source code repository while the pipeline and its components are built, tested, and packaged. In addition to building pipeline packages and pipeline executables, the CI process can perform unit testing on the engineering logic, test whether model training has converged, test each component in the pipeline to yield the desired artifact, test integration between pipeline components.

3. Pipeline continuous delivery: this module deploys the pipeline components generated by the continued integration phase to the target environment. The ML pipeline in production continuously provides new pipeline realization for a target environment, so that prediction service is provided for a new machine learning model trained by new feature data, and the output of the stage is a deployed machine learning pipeline;

4. pipeline automation module: the machine learning pipeline deployed in 3 is automatically executed in production, triggered by a schedule or actual demand. The output of this stage is the trained machine learning model, and the machine learning model is pushed to the model registry;

5. model continuous delivery: and taking the trained machine learning model as a prediction service for prediction. The output of this stage is the prediction service provided by the deployed machine learning model;

6. a monitoring module: the module collects statistical information about the performance of the machine learning model based on real-time data, the output of this stage being a trigger to execute a machine learning pipeline or to execute a new machine learning experiment cycle;

7. data and model validation module: the module automatically performs data verification and model verification after a new machine learning model is trained on new characteristic data so as to ensure that the new characteristic data is verified, the prediction quality of the machine learning model is evaluated, an evaluation metric value generated by the newly trained machine learning model is compared with the current machine learning model so as to ensure that the performance of the machine learning model is consistent on each part of data and ensure the basic structure compatibility of the machine learning model and the consistency of the machine learning model and a prediction service API;

8. a characteristic warehouse module: the module is a centralized repository in which the definition, storage and access of the standard characteristic data of the Xining model can be performed, so that the machine learning model training can be performed more quickly;

9. a metadata management module: the module records information about each execution of the ML pipeline, achieves reproducibility and comparison, and can also help debug errors and anomalies. Each time a pipeline is executed, the ML metadata store records start and end dates, time spent by the pipeline in each step, versions of the executed pipeline and components, model evaluation indexes generated for a training set and a test set in the model evaluation step, and other metadata;

10. a trigger module: the module implements automatic triggering of execution of the ML production pipeline to retrain the model using new data, e.g., monitoring of the process of performing predictive services on a target machine learning model deployed to a production environment, according to conditions set in advance; and when the preset abnormal service condition occurs in the process, re-executing the steps of creating and executing the continuous integration task through the continuous integration tool until the abnormal service condition does not occur in the process of executing the prediction service by the target machine learning model.

An embodiment of the present invention provides a computer device for implementing all or part of contents in the method for automatically delivering and deploying a machine learning model, where the computer device specifically includes the following contents:

a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between related devices; the computer device may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the computer device may be implemented with reference to the embodiment of the method for implementing automatic delivery and deployment of a machine learning model and the embodiment of the apparatus for implementing automatic delivery and deployment of a machine learning model in the embodiment, and the contents of the computer device are incorporated herein, and repeated details are not repeated here.

Fig. 10 is a schematic block diagram of a system configuration of a computer apparatus 1000 according to an embodiment of the present application. As shown in fig. 10, the computer apparatus 1000 may include a central processing unit 1001 and a memory 1002; the memory 1002 is coupled to the cpu 1001. Notably, this fig. 10 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.

In one embodiment, the automatic delivery and deployment of the machine learning model may be integrated into the central processor 1001. The cpu 1001 may be configured to perform the following control:

according to the source code, carrying out deployment pipeline processing and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model;

deploying the target machine learning model to a production environment, and performing performance testing;

when the performance test is failed, creating and executing a continuous integration task through a continuous integration tool until the target machine learning model passes the performance test;

the continuous integration task comprises:

continuously executing the steps of constructing the machine learning pipeline and constructing and training the machine learning model to be constructed through a continuous delivery tool to continuously obtain a target machine learning model;

and continuously deploying the obtained target machine learning model to the production environment through the continuous deployment tool.

In another embodiment, the device for automatic delivery and deployment of the machine learning model may be configured separately from the central processor 1001, for example, the device for automatic delivery and deployment of the machine learning model may be configured as a chip connected to the central processor 1001, and the function of automatic delivery and deployment of the machine learning model is realized by the control of the central processor.

As shown in fig. 10, the computer apparatus 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, a power supply 1007. It is noted that the computer device 1000 does not necessarily include all of the components shown in FIG. 10; furthermore, the computer device 1000 may also comprise components not shown in fig. 10, which can be referred to in the prior art.

As shown in fig. 10, the central processing unit 1001, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, and the central processing unit 1001 receives input and controls the operation of the various components of the computer apparatus 1000.

The memory 1002 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the cpu 1001 can execute the program stored in the memory 1002 to realize information storage or processing, or the like.

The input unit 1004 provides input to the cpu 1001. The input unit 1004 is, for example, a key or a touch input device. The power supply 1007 is used to supply power to the computer apparatus 1000. The display 1006 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.

The memory 1002 may be a solid state memory such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 1002 may also be some other type of device. Memory 1002 includes buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application/function storage part 1022, the application/function storage part 1022 being used for storing application programs and function programs or a flow for executing the operation of the computer device 1000 by the central processing unit 1001.

The memory 1002 may also include a data store 1023, the data store 1023 being used to store data such as contacts, digital data, pictures, sounds and/or any other data used by the computer device. Driver storage 1024 of memory 1002 may include various drivers for the computer device for communication functions and/or for performing other functions of the computer device (e.g., messaging applications, directory applications, etc.).

The communication module 1003 is a transmitter/receiver 1003 that transmits and receives signals via an antenna 1008. The communication module (transmitter/receiver) 1003 is coupled to the cpu 1001 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 1003, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same computer device. The communication module (transmitter/receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and receive audio input from the microphone 1010 to implement general telecommunications functions. The audio processor 1005 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 1005 is also coupled to the central processor 1001, so that sound can be recorded locally through the microphone 1010, and so that locally stored sound can be played through the speaker 1009.

In the embodiment of the invention, a source code for constructing a machine learning pipeline is developed according to the characteristic data of a machine learning model to be constructed; the machine learning pipeline is used for realizing the processes of constructing and training the machine learning model; according to the source code, carrying out deployment pipeline processing and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; constructing and training a machine learning model to be constructed according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain a target machine learning model; deploying the target machine learning model to a production environment to test the performance of the service; when the performance test is failed, creating and executing a continuous integration task through a continuous integration tool until the target machine learning model passes the performance test; the continuous integration task comprises: continuously executing the steps of constructing the machine learning pipeline and constructing and training a machine learning model to be constructed through a continuous delivery tool to continuously obtain a target machine learning model; compared with the technical scheme of manually constructing the machine learning model in the prior art, the method has the advantages that the constructed machine learning pipeline can automatically realize construction and training of the machine learning model, the problem that the traditional manual modeling process is too complicated in the prior art is solved, the machine learning model can be rapidly delivered and deployed, and the delivery and deployment efficiency of the machine learning model is improved; meanwhile, by using a continuous integration tool, a continuous delivery tool and a continuous deployment tool, automatic delivery, deployment and performance test of the machine learning model are realized, the problem that the traditional DevOps flow cannot be applied to machine learning model establishment in the prior art is solved, and better service can be provided for users.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for automatic delivery and deployment of a machine learning model, comprising:

2. The method of claim 1, wherein performing a persistent integration task, performing a deployment pipeline process, and constructing a machine learning pipeline comprises:

through deploying to the machine learning pipeline of target environment, according to the characteristic data, carry out the structure and the training of waiting to construct the machine learning model, obtain the target machine learning model, include:

3. The method of claim 1, further comprising:

testing and processing the constructed machine learning pipeline through a continuous integration tool; the test processing comprises: testing whether machine learning model training performed by the machine learning pipeline converges, whether the implementation effect of each pipeline component of the machine learning pipeline meets the preset requirement, and whether the integration degree among the testing pipeline components exceeds a preset value;

deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline, including:

4. The method of claim 1, further comprising:

and carrying out model registration on the target machine learning model.

5. The method of claim 1, further comprising:

6. The method of claim 1, further comprising:

adding version identification for each constructed machine learning pipeline;

7. The method of claim 1, further comprising:

8. An apparatus for automatic delivery and deployment of a machine learning model, comprising:

the machine learning pipeline building module is used for creating a continuous integration task according to the source code; executing a continuous integration task, performing deployment pipeline processing, and constructing a machine learning pipeline; deploying the machine learning pipeline to a target environment corresponding to the machine learning pipeline; the continuous integration task comprises: continuously executing the step of constructing the machine learning pipeline through a continuous delivery tool to obtain the machine learning pipeline; continuously deploying the obtained machine learning pipeline to a target environment corresponding to the machine learning pipeline through a continuous deployment tool;

the target machine learning model building module is used for building and training a machine learning model to be built according to the characteristic data through a machine learning pipeline deployed to a target environment to obtain the target machine learning model;

9. The apparatus of claim 8, wherein the machine learning pipeline building module is specifically to:

10. The apparatus of claim 8, further comprising:

test processing module for

the machine learning pipeline building module is specifically used for:

11. The apparatus of claim 8, further comprising:

a model registration module to:

and carrying out model registration on the target machine learning model.

12. The apparatus of claim 8, further comprising:

a feature data acquisition module to:

13. The apparatus of claim 8, further comprising:

a recording module to:

adding version identification for each constructed machine learning pipeline;

14. The apparatus of claim 8, further comprising:

a service anomaly monitoring module for:

15. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 7 when executing the computer program.

16. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.

17. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 7.

18. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 7.